Relative Touch User Interface Enhancements

ABSTRACT

Some embodiments provide a meta touch interface (MTI) with multiple position indicators with each position indicator operating as a separate pointing tool that can be activated (i) using taps on a touchpad or other touch sensitive surface or (ii) by pressing certain keyboard keys. The MT pointer allows for adjacent UI elements to be selected without having to reposition the MT pointer for each selection or activation. Some embodiments provide a multi-device UI that comprises at least two UIs, wherein the first UI is presented on an essentially horizontal plane that is aligned with operational focus and the second UI that is presented on an essentially vertical plane that is aligned with visual focus of the user. Some embodiments provide a precision pointer that includes an adjustable magnified region to better present underlying on-screen content, thereby allowing the user to more precisely position the pointer.

CLAIM OF BENEFIT TO RELATED APPLICATIONS

This application is a continuation of U.S. nonprovisional applicationSer. No. 14/595,023, entitled “Relative Touch User InterfaceEnhancements”, filed Jan. 12, 2015 which is a continuation of U.S.nonprovisional application Ser. No. 13/490,113, entitled “Relative TouchUser Interface Enhancements”, filed Jun. 6, 2012, now U.S. Pat. No.8,933,888, which claims the benefit of U.S. provisional application61/493,523, entitled “Enhanced Pointer Tools for Graphical UserInterfaces”, filed Jun. 6, 2011, is a continuation-in-part of U.S.non-provisional patent application Ser. No. 13/423,270 entitled “InputDevice User Interface Enhancements” filed on Mar. 18, 2012, now U.S.Pat. No. 9,195,321, which claims the benefit of U.S. provisionalapplication 61/465,345, entitled “Input Device User InterfaceEnhancements” filed Mar. 17, 2011, and is a continuation-in-part of U.S.non-provisional patent application Ser. No. 13/423,212 entitled “TouchEnhanced Interface” filed Mar. 17, 2012, now U.S. Pat. No. 8,760,424,which claims the benefit of U.S. provisional application 61/465,309,entitled “Touch Enhanced Interface”, filed Mar. 17, 2011. The contentsof provisional applications 61/493,523, 61/465,345, and 61/465,309 andnon-provisional application Ser. Nos. 14/595,023, 13/490,113, 13/423,270and Ser. No. 13/423,212 are hereby incorporated by reference.

TECHNICAL FIELD

The present invention generally relates to software and hardware userinterface enhancements for input devices on computer systems andelectronic devices.

BACKGROUND ART

The modern operating system (OS) provides a graphical user interface(GUI) for interacting with the user interface (UI) elements presenttherein. The GUI makes use of a spatial selection paradigm in whichnavigating the GUI involves mapping movements that are detected usingsome input device (e.g., a mouse) to the virtual interface of the GUI inorder to interact with the user interface (UI) elements of the GUI. Ascomputers and input devices have evolved from keyboards and mice tomodern day multi-touch trackpads and multi-touch touchscreen surfaces,the GUI has been adapted to maintain compatibility with legacy inputdevices (i.e., keyboard and mouse) while also incorporating benefitsthat are provided by the newer input devices. In so doing however, theOS and/or GUI has mutated in a way such that no single input device orinput paradigm is singularly suited to fully interact with the OS and/orGUI because each input device has its own set of advantages anddisadvantages.

The keyboard is a discrete input device wherein each key of the keyboardcan be used to provide a discrete input. The keyboard is efficient foruse in performing text entry, but inefficient for use in navigating thespatial selection paradigm of the GUI. While keyboard “shortcuts” whichutilize “flag keys (e.g., option, control, command, alt, etc.) provideaccess to certain elements of a GUI, such keyboard shortcuts are oftendifficult to remember, inconsistent across different applications, andto invoke a keyboard shortcut, the user must often perform awkward handmovements and finger extensions in order to reach flag keys that areused in combination with other keys. As a result, the keyboard remainsthe preferred input device when performing operations that requirediscrete input (e.g., text entry), but is deficient when navigating andutilizing other aspects of the GUI.

In direct contrast to the keyboard, pointer based input devices, such asa mouse or trackpad, are useful for navigating a GUI and selecting andinvoking user interface (UI) elements therein, but are inefficient foruse in performing text entry. Consequently, to operate a modern daycomputer system or device, many users rely on the functionality of akeyboard used in conjunction with a pointer based input device.

Touchscreen based input devices have attempted to bridge the gap betweendiscrete input devices and spatially dependent input devices.Specifically, touchscreen based input devices allow users to directlytouch and thereby manipulate UI elements that are perceived on thedevice, which provides much of the traditional pointer basedfunctionality. To then function as a keyboard, devices that utilizetouchscreen based interaction provide virtual keyboards that aredisplayed on-screen in conjunction with various touch based gesturesthat assume the role of text entry. However, for intensive textcomposition, coding, game playing, or other common computer activitieswhere discrete and/or precise input is required, a secondary inputdevice such as the keyboard is often used to overcome the deficienciesand lack of tactical feedback of the touchscreen based input device.Furthermore, precise manipulation is challenging on touch based devices,because the user's fingers necessarily obstruct the exact area where themanipulation is desired. Also, there are ergonomic limitations toprolonged usage when either the arm is held up to the screen, or theneck is strained by looking down at the screen.

Accordingly, there is a need to better adapt existing inputfunctionality of one or more of these input devices to bridge the dividein navigating a GUI and performing text entry and other input thatrequires precision or discrete interaction. Stated differently, there isa need to integrate functionality from pointer based UIs, direct touchUIs, and discrete input UIs to update the capabilities of the GUI andmodern day OS software to better adapt to various input device hardwareadvancements.

SUMMARY

It is an objective of the present invention to define an apparatus,method, system, and computer software product that integratesfunctionality from pointer based user interfaces (UIs), direct touchUIs, and discrete input UIs to enhance traditional graphical userinterface (GUI) pointing tool interactions. It is further an objectiveto permit and encourages the device user to retain visual focus on thedisplay device(s) on a vertical plane and manual touch on the inputdevice(s) on a horizontal plane. To achieve these and other objectives,some embodiments provide a meta touch interface (MTI) for facilitating arelative touch UI which approximates and expands upon a screen-mappeddirect-touch UI. The MTI includes a graphical representation withmultiple position indicators with each position indicator operating as aseparate pointing tool that can be activated (i) using taps on atouchpad or other touch sensitive surface or (ii) by tapping certainkeyboard keys. Activating a position indicator includes performing amouse left-click, right-click, hold, drag, or double-click as someexamples. The position indicators move as a group in conjunction withmovements of the MT pointer. The MT pointer allows for related sets ofUI elements that are often located adjacent to each other on-screen tobe manipulated without having to reposition the MT pointer prior to eachselection, activation or manipulation.

The configuration of the position indicators can be adjusted toreposition each indicator relative to the other indicators and to adjustthe size of the indicators. In some embodiments, the configuration ofthe position indicators is automatically adjusted so that the positionindicators are better positioned in orientation and distance with theposition of the UI elements on-screen. In some embodiments, an autosnapping feature is provided whereby the position indicators areautomatically placed over UI elements that are adjacent or nearby andthat can be selected. Accordingly, some embodiments provide a UI elementdisambiguation function to automatically identify the adjacent orneighboring UI elements before snapping the position indicators of theMT pointer to those UI elements.

It is a further objective to provide the user differing sets offunctionality that can be enabled using different finger combinationtouch articulation by the user. Accordingly, some embodiments enable afirst set of functionality to manipulate a pointed to UI element basedon a touch action of the index finger, a second set of functionality tomanipulate the pointed to UI element based on a touch action of themiddle finger, a third set of functionality to manipulate the pointed toUI element based on a touch action of the ring finger, and a fourth setof functionality to manipulate the pointed to UI element based on atouch action of the little finger. Further sets of functionality arebased on touch actions of combinations of two or more digits, includingthe thumb. The touch action involves issuing a touch and hold on atrackpad or touchscreen surface with the corresponding digit or digits.Once a specific set of functionality is enabled, the other digit ordigits not used to enable the set of functionality can be used to invokeor otherwise activate the enabled functions.

It is a further objective to provide the user with visual aids and othertools which enable the brain to make a strong correspondence betweenhand/finger motions and resulting computer operations, in order tominimize or eliminate the need for the user to constantly shift manualand visual focus. To achieve these and other objectives, someembodiments provide a multi-device UI that comprises at least two UIs,wherein a first UI is touch enabled and the first UI displays at least asubset of the second UI. The first UI may be presented using atouchscreen tablet, touchscreen smartphone, or other touch sensitivedisplay on an essentially horizontal plane that is synchronously coupledwith the second UI that may be presented using a monitor or displaydevice that may be placed farther from the user on an essentiallyvertical plane. Synchronously coupling the first UI to the second UIallows touch inputs issued using the first UI to produce change on thesecond UI.

It is further an objective of the present invention to define anapparatus, method, system, and computer software product that overcomeslimitations and enhances the use of the human finger as a UI pointingdevice on touchscreen and trackpad devices. Specifically, someembodiments provide a precision pointing function that seamlesslyoperates with traditional trackpad and/or touchscreen functionality. Theprecision pointer includes a configurable magnified region to betterpresent underlying on-screen content. The level of magnification adjuststhe speed of movement over the underlying content and with the magnifiedview of the underlying content, the user is able to more preciselyposition the pointer. For a touchscreen, the magnified region ispresented at the edge of the user's touch such that the user's touchdoes not obscure the underlying content.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to achieve a better understanding of the nature of the presentinvention and the objectives previously enumerated, preferredembodiments of the Meta Touch Interface (MTI) and operation of the MetaTouch pointer will now be described, by way of example only, withreference to the accompanying drawings in which:

FIG. 1 illustrates the Meta Touch Interface (MTI) in accordance withsome embodiments.

FIG. 2 conceptually illustrates how each digit indicator of the MTpointer essentially represents a different pointing tool that can beused to independently access different UI elements.

FIG. 3A illustrates using the MT pointer to operate a set of mediaplaying controls.

FIG. 3B illustrates using the MT pointer to access related functionalityof a calculator application.

FIG. 4A provides examples of the MT pointer snap functionality of someembodiments with the MT pointer represented in six positions proximateto a set of grouped controls.

FIG. 4B details the process used by some embodiments to determine howthe MT pointer snaps to grouped controls.

FIG. 5 illustrates some other UI element groupings for which the MTpointer is well suited.

FIG. 6 conceptually illustrates how the touch of the left hand thumbcontact area differs from the right hand thumb contact area, and how itdiffers from a touch contact area of the index, middle, ring, or littlefingers.

FIG. 7 illustrates the advantage of being able to use different digitindicators of the MT pointer to access different UI elements withouthaving to reposition the MT pointer.

FIG. 8 illustrates performing a four touch slide gesture to increase thesize of the MT pointer and its digit indicators in accordance with someembodiments.

FIG. 9 illustrates performing a four finger touch and subsequentlymoving the touch of only the index finger to adjust the relativeposition of the index finger digit indicator of the MT pointer inaccordance with some embodiments.

FIG. 10 illustrates performing a five finger expansion gesture toreposition each of the digit indicators of the MT pointer.

FIG. 11 illustrates an embodiment whereby a standard keyboard is used tooperate the MT pointer.

FIGS. 12-14 illustrate a single touch sensitive key embodiment foroperating MT pointer.

FIG. 15 illustrates adapting the touch surface of the touch sensitivespacebar keyboard key to provide different functionality depending onwhere a touch or tap is applied to the spacebar.

FIG. 16 illustrates usage of the enhanced keyboard with fingers of onehand providing touches and gestures on the spacebar where such touchesand gestures can be used to emulate traditional trackpad functionality,move the MT pointer when the MTI is enabled, and activate the variousdigit indicators of the MT pointer when the MTI is enabled.

FIG. 17A conceptually illustrates an internal mathematical model fortracking an on-screen position of the MT pointer in accordance with someembodiments.

FIG. 17B illustrates MT pointer movement in an embodiment whichprocesses touch data from a device such as a trackpad which tracksmultiple digits.

FIG. 18A presents a proximity assignment process performed by the MTI todetermine which touch corresponds with which finger in accordance withsome embodiments.

FIG. 18B provides several examples of shift assignments which maptouches to fingers without regard to previous touch assignmentlocations.

FIG. 18C provides a process used in some embodiments to make assignmentdeterminations when finger, thumb and resting touches are considered.

FIG. 19 illustrates hotzones that are associated with each digitindicator of the MT pointer in accordance with some embodiments.

FIG. 20 presents the disambiguation process that is performed by the MTIin accordance with some embodiments.

FIG. 21 illustrates an inset region and an outset region for arectangular shaped UI element.

FIG. 22 illustrates when the hotspot of a particular digit indicatorfalls within an inset region of a UI element, that particular digitindicator is unambiguously positioned over that UI element.

FIG. 23 illustrates when the hotspot of a particular digit indicatorfalls within an outset region of only a single UI element, thatparticular digit indicator is unambiguously positioned over that UIelement.

FIG. 24 illustrates when the hotspot of a particular digit indicatorfalls within the outset regions of at least two UI elements, then thatparticular digit indicator is ambiguously positioned and thereforecannot be used to activate either of the UI elements.

FIG. 25 illustrates when the hotspot of a particular digit indicatorfalls within the outset region of a first UI element and within theinset region of a second UI element, then that particular digitindicator is unambiguously positioned over the second UI element and canbe used to activate the second UI element.

FIG. 26 illustrates the visual identifiers in accordance with someembodiments.

FIG. 27 illustrates the MTI auto-snap function at a first y-axial planeof the GUI.

FIG. 28 illustrates the MTI auto-snap function at a second y-axial planeof the GUI.

FIG. 29 illustrates the MTI auto-snap function at a third y-axial planeof the GUI.

FIG. 30 illustrates the position of the digit indicators along they-axial plane of FIG. 29 after the digit indicators were moved twoincremental positions to the right.

FIG. 31 presents a process for performing the auto-snap MTIfunctionality in accordance with some embodiments.

FIG. 32A illustrates various hold action functionality in accordancewith some embodiments.

FIG. 32B provides an example of both location specific and omni-touchfunctionality while performing the common task of selecting some text,and then moving that text to another location within the paragraph.

FIG. 32C details the process used to determine various hold action modesand operations.

FIG. 32D provides further detail on how these tap-hold actions areinvoked and operated.

FIG. 32E depicts a simplified HUD of the MTI digit display in accordancewith some embodiments.

FIG. 33 illustrates single-pointer mode MTI functionality in accordancewith some embodiments.

FIG. 34A presents a usage example of the synchronized UIs in accordancewith some embodiments.

FIG. 34B presents an alternative synchronized multi-device embodimentwherein the horizontal device is used to control various operations onthe vertical device.

FIG. 34C shows the process used by some embodiments to implement thesynchronized multi-device functionality of FIG. 34B.

FIG. 35A illustrates an alternate usage example of the synchronized UIsin accordance with some embodiments.

FIG. 35B presents an alternative synchronized multi-device embodimentthat correlates with the embodiment presented in FIG. 34B above.

FIG. 36A illustrates still another alternate usage example of thesynchronized UIs in accordance with some embodiments.

FIG. 36B presents an alternative synchronized multi-device embodimentwherein the screen graphics of the horizontal device are sent to thevertical device so that they can be used for reference.

FIG. 37 illustrates the precision pointer in accordance with someembodiments.

FIG. 38 illustrates increasing the magnification of the precisionpointer using an expansion gesture in accordance with some embodiments.

FIG. 39 presents a process for implementing the precision pointer inaccordance with some embodiments.

FIG. 40 illustrates a computer system with which some embodiments areimplemented.

DETAILED DESCRIPTION

I. Meta Touch Interface

To address the previously discussed limitations associated with variouscomputer input devices, it is an objective of the present invention todefine an apparatus, method, system, and computer software product thatintegrates functionality from pointer based UIs, direct touch UIs, anddiscrete input UIs to enhance traditional GUI pointing toolinteractions. It is further an objective of the present invention todefine an apparatus, method, system, and computer software product thatpermits and encourages the device user to retain visual focus on thedisplay device(s) on a vertical plane at eye level and manual touch onthe input device(s) on a horizontal plane at hand level. To do so, someembodiments provide a meta touch interface (MTI) which simulates ascreen-mapped direct-touch UI by allowing physical control to beperformed on one or more separate, more conveniently situated surfaces,with analogous movements represented in context on-screen by way of aheads-up interactive meta touch pointer display. It should be noted thatthe Meta Touch Interface (MTI) and the MT pointer were referred to asthe Hand Pointer Interface (HPI) and the HPI pointer in the provisionalpatent application 61/493,523 to which the present application claimsbenefit.

FIG. 1 illustrates the Meta Touch Interface (MTI) in relation to atraditional pointer-based interface. The MTI includes an interactivegraphical pointer with multiple digit and position indicators.Specifically, the graphical representation is superimposed as apartially transparent graphical hand overlay, also referred to as themeta touch (MT) pointer 110, with five digit indicators 130, 140, 150,160, and 170 that correspond to the thumb, index finger, middle finger,ring finger, and little finger of either the left hand, the right hand,or both. For the sake of clarity and brevity, most figures anddescriptions will refer to a single right hand representation, thoughthis is not meant to limit the present disclosure in any way. Therepresentations and operation of the MTI, MT pointer, and enhancementsare applicable to right-handed, left-handed, or dual-handed operationand such operation is interchangeable depending on the preferences ofthe user.

In some embodiments, each digit indicator 130-170 operates as a separatepointing tool that can be activated (i) using taps on a touchpad orother touch sensitive surface or (ii) by tapping certain keyboard keyssuch as the “home” keys. Specifically, the index finger digit indicator140 can be activated using a tap of the index finger on a trackpad orother touch sensitive surface or by tapping the “J” keyboard key; themiddle finger digit indicator 150 can be activated using a tap of themiddle finger on a trackpad or other touch sensitive surface or bytapping the “K” keyboard key; the ring finger digit indicator 160 can beactivated using a tap of the ring finger on a trackpad or other touchsensitive surface or by tapping the “L” keyboard key; and the littlefinger digit indicator 170 can be activated using a tap of the littlefinger on a trackpad or other touch sensitive surface or by tapping the“Semicolon” keyboard key. Detecting which finger corresponds with whichdetected tap is described below. Activating or manipulating a digitindicator includes performing a mouse left-click, right-click, hold,drag, or double-click as some examples. The digit indicators 130-170move as a group in conjunction with movements of the MT pointer 110.However, MTI functionality allows a user to adjust the size of the digitindicators 130-170 or independently position each digit indicatorrelative to the MT pointer 110 as a whole. FIG. 2 conceptuallyillustrates how each digit indicator of the MT pointer 110 essentiallyrepresents a different pointing tool that can be used to independentlyaccess different UI elements wherein accessing a UI element includesselecting the UI element, invoking an output action associated with theUI element, or invoking alternate functionality (i.e., right-clicking,moving, or otherwise manipulating the UI element).

The MT pointer 110 replaces or otherwise compliments traditional GUIpointing tools, such as the traditional GUI pointing tool 120. As iswell known in the art, a traditional GUI pointing tool 120 may be usedto manipulate a single UI element at a time, before repositioning thepointing tool. For each UI element access, the pointing tool 120 ismoved to the corresponding on-screen location of that UI element priorto access. The MT pointer 110 overcomes the traditional pointing tool's120 inability to access multiple UI elements from a single on-screenposition. More specifically, the MT pointer 110 allows, among otherthings, related sets of UI elements that are often located adjacent toeach other on-screen to be activated without having to reposition the MTpointer 110 for each such activation. For example, a media playerapplication will often have previous track, play/pause, and next trackUI elements placed adjacent to each other. Using the MTI, the MT pointerneed only be positioned once in order to easily and reliably access eachof the three UI elements. Specifically, the MT pointer is positionedsuch that three of the digit indicators are positioned over the three UIelements. Then, by activating each of those three digit indicators, theuser can access all three UI elements without having to reposition theMT pointer. This is particularly helpful when the user must oftenvacillate between related controls. The user thus spends less timeprecisely positioning a single pointer tool to navigate the GUI in orderto access the desired functionality. Importantly, the MTI also allowssimultaneous operation of UI elements by pressing various combinationsof fingers. This multiplies the usefulness of many control groupings, aswill be detailed below. In some embodiments, the MT pointer and thecorresponding digit indicators are semi-transparent so as to not obscureUI elements and other content when the MT pointer is positioned over oneor more UI elements and other content. In some embodiments, the digitindicators include a fully transparent circular area at the tip of eachdigit indicator that represents the area of the digit indicator at whicha UI element can be accessed. This fully transparent area allows betterviewing of the accessible UI element or other content. Other visualeffects may be provided to distinguish the MT pointer and digitindicators from underlying content, such as shadows or specificcompositing techniques, as well as increasing the opacity as the MTpointer moves, so that the user can more readily track the MT pointerwhile in motion.

A. Usage

Various examples illustrating the use cases of the MT pointer and itsadvantages over a traditional GUI pointing tool are presented below. Thescope of the present invention is not intended to be limited to thespecific examples given. In light of present description, it should beapparent that other use cases and advantages are applicable, though notexplicitly described herein.

One advantage of the MT pointer over traditional single pointing toolsis exemplified with adjacent UI elements that are frequently accessed orthat provide access to related functionality. FIG. 3 illustrates usingthe MT pointer 310 to operate a set of media playing controls. In thisfigure, the MT pointer 310 is positioned over media playing controls.Specifically, the MT pointer 310 is positioned with the index fingerdigit indicator 320 over the fast reverse or previous track UI element,the middle finger digit indicator 330 over the play/pause UI element,and the ring finger digit indicator 340 over the fast forward or nexttrack UI element. These UI elements provide access to commonly usedfunctions of a media playing application. Accordingly, the user willoften access these UI elements collectively to select media to play, toadjust a position in the selected media, pause the playback, and changethe media selection. By placing the MT pointer 310 over these UIelements, the user can reliably access corresponding functionality ofeach of the three UI elements at any time without repositioning of theMT pointer 310. In other words, the user can skip tracks, pause or playa track, and forward or reverse in a track without having to repositionthe MT pointer 310. As an example, the user may tap the middle finger topause playback, and then quickly tap the ring finger several times tomove through the next selections. Due to system lag time or otherfactors, the selection may proceed beyond the desired track, and theuser must then use the previous track button, which can be done easilyby tapping the index finger. When the correct track is identified, theuser taps the middle finger to start playback. At this point the usermay press and hold the ring finger to fast forward to a particularmusical passage. Once again, any overshoot in the playback position maybe corrected by pressing the opposing rewind control on the indexfinger. The process of iteratively adjusting values which wasillustrated twice in the preceding example is an extremely common taskin many applications. The MTI of the present invention makes such tasksfar more efficient and ergonomically palatable by obviating the need tore-position a pointer between each such adjustment.

Furthermore, the MT pointer allows for simultaneous control of UIelements within a grouping. For example, if the user activates theforward scan button by pressing and holding the ring finger 340, theuser may also press the opposing control 320 to double the scan speed.Likewise, while holding down the ring finger 340, the user mayintermittently press the middle finger 330 to reduce the speed of theaudio scanning operation for easier identification. This type ofsimultaneous operation provided by the MT pointer is beneficial incountless scenarios where continuous control is desired, from UIelements as a simple as a volume control to 3d game play.

As the user moves the MT pointer in the vicinity of specifically groupedsets of controls, the MT pointer snaps to the preferred configuration ofdigit spacing and assignment. This allows for fast and consistentplacement of the MT pointer, and ensures that control of specificfunctions is always mapped to specific fingers. This allows musclememory to be built automatically with each use. FIG. 4A providesexamples of the MT pointer snap functionality of some embodiments withthe MT pointer represented in six positions proximate to a set ofgrouped controls. The grouping of four related controls 410 a, 420 a,430 a, 440 a, 450 a and 460 a includes (from left to right) a stopbutton, a play button, a rewind button and a fast forward button.Because MT pointer 415 a is not close enough to the grouped controls 410a, the grouped controls 410 a are not highlighted. When the user movesthe MT pointer 425 a closer to the grouped controls 420 a, they arehighlighted in some way to inform or remind the user that the controlsare operable as a group. In some embodiments, the highlighting utilizesone or more graphical techniques (e.g., color coding, link lines) todepict the one-to-one correspondence of each digit with the specificcontrol within the grouping. When the user releases the mouse button (ortouch, etc.) to stop moving the MT pointer, the MT pointer automaticallymoves to an ideal position 435 a in which the digits are approximatelycentered on the corresponding controls 430 a, and the highlightingeither fades away or changes to some other form of highlighting thatindicates to the user that the pointer is properly snapped in place.When the MT pointer 445 a and 455 a is positioned by the user such thatthe digit correspondence is improper, the MT pointer will automaticallymove to the ideal position 435 a if the user concludes the moveoperation at this position. In some embodiments, the user may also flickthe MT pointer toward the direction of the grouped controls to snap theMT pointer to the grouped controls in the ideal position 435 a. If theuser concludes the drag operation with the MT pointer 465 a not inimmediate proximity to the grouped controls 460 a, any highlighting isremoved.

FIG. 4B details the process used by some embodiments to determine howthe MT pointer snaps to grouped controls. When the MT pointer is beingdragged to a new position 410 b, the process determines if there are anygrouped control UI elements close to any digits of the MT pointer, andif so, calculates the distance to the closest such UI element 420 b. Insome embodiments, group controllable UI elements are determined inadvance by system API calls, and the leftmost and rightmost positions ofeach grouped control set is stored for subsequent operations. Then, asthe MT pointer is moved, proximities to the new MT pointer position iscalculated from the leftmost MT pointer digit position to the rightmostgrouped control position, and the rightmost MT pointer digit position tothe leftmost grouped control position. If the closer of the twoproximities is within a small range (e.g., 40 pixels), the groupedcontrols are highlighted 430 b in some way. Some embodiments highlightthe group as a whole (e.g., a darker shading behind the grouping),highlight individual group elements (e.g., a thicker border around eachUI element), and/or highlight digits of the MT pointer (e.g., colorcoding or link lines to the corresponding UI element). If the usercontinues to drag the MT pointer, the process continues 410 bdetermining proximities and highlighting or un-highlighting groupcontrol UI elements. Otherwise, if the user has concluded the dragoperation 440 b, the MT pointer is moved to the ideal position 450 b ifit is within the determined proximity. In some embodiments, the MTpointer is moved to the ideal position as a whole (without adjustinginter-digit spacing) so that it is as closely centered on the UIelements as possible. In other embodiments, the position of each digitis matched with the exact or proximate location of the matching UIelement (within a range), and when the pointer is later moved theinter-digit spacing is restored. With the MT pointer and its digits areideally positioned, the grouped control highlighting fades away 460 b oris altered in some way, and the process continues 410 b. In someembodiments, grouped controls are determined in advance by the softwarecreator, so that the process can determine which UI elements should beconsidered as grouped controls. Furthermore, if the user positions theMT pointer over non-grouped controls and then activates various UIelements with specific digits of the MT pointer, this correspondence ofdigit activation per UI element is stored in the MTI database. Afterseveral such consistent usages, the MTI system determines that the soidentified adjacent UI elements should be considered as a grouped set.The user would then see the grouped highlighting and snapping while inoperation, and muscle memory can be formed because a particular fingeris always used to trigger a particular UI element.

FIG. 3B illustrates using the MT pointer 350 to access relatedfunctionality of a calculator application. In this figure, the MTpointer 350 is positioned over a row of UI elements that representnumeric keys of a calculator. Specifically, the MT pointer 350 ispositioned with the index finger digit indicator 360 over the “4”numeric key, the middle finger digit indicator 370 over the “5” numerickey, and the ring finger digit indicator 380 over the “6” numeric key.Once positioned over this set of keys, the user can access each of theseUI elements (e.g., numbers “4”, “5”, and “6”) without further movementof the MT pointer. As shown, the user activates the index finger digitindicator 360 to enter the number “4” and then the user activates themiddle finger digit indicator 370 to enter the number “5”. Activatingthe various digit indicators can also be performed using discretekeyboard keys or taps on a touchpad as will be explained in greaterdetail below. In comparison, when using a traditional single pointingtool, the user first positions the pointing tool over the “4” UIelement, performs a left click to enter the number “4”, then repositionsthe pointing tool over the “5” UI element, and performs a left click toenter the number “5”. By using the MT pointer 350 in this simplifiedexample, the second positioning step is avoided. To access othernumbers, MT pointer movements are simplified because the user only needsto move the pointing device in the vertical plane, as the x-axis remainsconstant. As previously mentioned, the MTI of some embodimentsincorporates a “snap” algorithm which senses grouped controls andfacilitates this kind of single-axis or otherwise simplified pointermovement. Additionally, some embodiments provide tap above and belowfunctionality to access, for instance, the “7” and “1” buttons with theindex finger, without repositioning the MT pointer at all. Furthermore,when the digit indicators are activated by discrete keyboard keys (aswill be described below), keys above and below the normal “home”activating keys may also be used to access UI elements above and belowthe digit indicators in some embodiments.

Because the MPI system can distinguish thumb touches from fingertouches, some embodiments utilize the thumb to move the MP pointer andthe fingers to activate various UI elements over which they arepositioned. For example, in FIG. 4, the thumb is used to move the MPpointer by simply resting the thumb on the trackpad (or other controlsurface) and dragging. Any UI element that is beneath the thumb pointeris not clicked or activated in any way, which allows the user to dragthe MT pointer into position without being concerned about accidentalactivation. After positioning, the fingers are free to simply tap on thedesired calculator buttons with the appropriate digit.

FIG. 5 illustrates some other UI element groupings for which the MTpointer is well suited. These UI groupings are shown in a typical mediacontrol surface which includes a group of transport controls 505(previously discussed), volume slider 510, and a grouped set of viewselector UI elements 520. The MT pointer may be used to access theslider UI element 510, because when the MT pointer is positioned overthe slider 510 near the position indicated by the arrow, differentpositions of the slider UI element 510 can be accessed based on thepositions of the different digit indicators of the MT pointer relativeto the slider UI element 510. The slider UI element 510 can control avolume setting as shown, or any number of other value settingparameters, such as a track position slider in a media playingapplication. The view controller 520 of the media player applicationselects one of four ways to view the track data (not shown in figure).As before, when the MT pointer index finger digit indicator is placed onthe leftmost element, it is easy for the user to switch betweendifferent views. This quick, intuitive access encourages the user tomore fully utilize the features of the application because it is easierto simply tap a finger and see the results, rather than position thecursor on each and read “help text” or try to remember the meaning of asmall icon with abstract graphics.

530 depicts a rotary controller for specifying a value based onindividual component values. In addition to activating a UI elementidentified by a digit indicator by tapping, values may be adjusted foreach segment of the controller by utilizing drag, swipe and holdgestures on the corresponding touch or key. Standard features such asinertial scrolling and stopping can thus be utilized withoutrepositioning the MT pointer, and simultaneous control of two or moreparameters can be effectuated. Similarly, manipulating multipleindividual related controls simultaneously is shown in an audio mixingapplication 540, where such control is an important component ofsuccessful use. In this setting, while a thumb touch is maintained, theMT pointer remains stationary, allowing the fingers to manipulate theircorresponding UI objects without moving other digits of the MT pointeroff of their intended UI elements.

Operation of the MT pointer 110 is discussed below in relation to atrackpad, trackpad and keyboard combination, and keyboard withintegrated trackpad functionality.

B. MTI Operation with Trackpad

In some embodiments, the MTI is controllable using a trackpad or othersimilar multiple touch detecting input device. In this method of MTIoperation, on-screen movement of the MT pointer, including all fivedigit indicators, is controlled by single touch movements on thetrackpad, or multiple simultaneous touch movements that are essentiallyparallel, in a manner similar to how a single touch movement produces acorresponding movement of a traditional pointing tool. When the MTI isactive, instead of moving the traditional pointing tool, the MTI adaptstouch inputs that are provided by the trackpad to perform movements ofthe MT pointer. The MT pointer can be moved anywhere on-screen so longas at least some part of a digit indicator is positioned within thescreen bounds. For instance, the MT pointer can be positioned such thatthe thumb digit indicator is moved off-screen and the other digitindicators remain active. Such functionality also applies when, forexample, a touch of the thumb is extended beyond a touch surface of atrackpad or touchscreen to a touch sensitive bezel or border of thetrackpad or touchscreen.

In some embodiments, touch movements of the thumb are used to move theMT pointer, and the fingers are used to activate functionality after theMT pointer is in place. The thumb is a preferred manner to move theon-screen position of the MT pointer, because it frees the fingers toactivate functionality by a tap, hold, drag, flick, other touch, orgesture. Because the thumb is used for movement only, and a tap of thethumb does not activate anything that happens to be beneath theon-screen location of the thumb, there is no consequence to the userinadvertently tapping the thumb while making repeated MT pointermovements with the thumb. Also, a detected touch of the thumb may bedistinguishable on a trackpad from detected touches of other digits.Moreover, a touch of the right hand thumb is distinguishable from atouch of the left hand thumb and this allows for the MT pointer tochange from a right hand representation to a left hand representation.In some embodiments, the user signals a switch between left and righthands by tapping all 5 digits on the trackpad. FIG. 6 conceptuallyillustrates how the touch of the left hand thumb contact area 610differs from the right hand thumb contact area 620, and how it differsfrom a touch contact area 630 of the index, middle, ring, or littlefingers. In some embodiments, the MTI analyzes the touch contact areageometry data from the trackpad to identify a touch of the left handthumb, the right hand thumb, and touches of the other fingers based onthe overall perceived shape of the contact area. In some embodiments,data from the use of the device is analyzed and stored to refine andimprove the accuracy of thumb/finger differentiation. In otherembodiments the trackpad itself and/or the system software may beresponsible for making such distinctions and pass such information tothe MTI engine in a more concise manner. Though the discussion abovepertains to movements of the MT pointer using the thumb, other means arealso available to move the MT pointer including using a finger, orparallel movements of two or more fingers in some embodiments.

As noted above, each digit indicator of the MT pointer can beindependently activated based on different user inputs. For trackpadoperation, touches and taps performed by different fingers of the user'shand are used to independently activate each of the MT pointer's digitindicators. In other words, each finger of the user's hand activates adifferent MT pointer digit indicator. A single tap performs atraditional single mouse left-click, a double tap performs a traditionalmouse double-click, and a press and hold performs a traditionalmouse-down message followed by a mouse-up message when the touch isreleased. If the touch is moved while held down, traditional mouse-dragmessages are sent. This is in contrast to current art where a tapgesture is typically actuated with a single finger and a hold or move ordrag operation requires two fingers or a switch depression or some otherdistinguishing characteristic. In some embodiments, other messages suchas a mouse right-click are performed when the user performs a differenttype of touch with the corresponding digit indicator. Other usageexamples include performing a slide or flick up to scroll a view up anda slide or flick down to scroll a view down in some embodiments. The MTIsupports other touches and combinations of touches such as an examplepresented below which utilizes a press and hold of a first digit and asubsequent tap of a second digit.

FIG. 7 illustrates the advantage of being able to use different digitindicators of the MT pointer to access different UI elements withouthaving to reposition the MT pointer. In this figure, the MT pointer 710has digit indicators 720, 730, and 740 positioned over UI elements 750,760, and 770. The digit indicator 720 corresponds to the index finger ofthe user's right hand, the digit indicator 730 corresponds to the middlefinger of the user's right hand, and the digit indicator 740 correspondsto the ring finger of the user's right hand. Therefore, to access UIelement 750, the user taps the index finger on the trackpad to activatethe index finger digit indicator 720 of the MT pointer 710. Similarly,to access UI element 760, the user taps the middle finger on thetrackpad to activate the middle finger digit indicator 730 of the MTpointer 710, and to access UI element 770, and the user taps the ringfinger on the trackpad to activate the ring finger digit indicator 740of the MT pointer 710. Various techniques described below in theimplementation section describe how the MTI differentiates betweentouches of the index, middle, ring, and little fingers on the trackpad.Not shown in FIG. 7 is the effect of the tap on the UI elements, whichchange the calendar view from day to week to month. As noted in theusage examples in section A above, this permits the user to quicklychange the view back and forth without repositioning the MT pointer.

In some embodiments, the MT pointer is automatically scaled to a sizethat is optimized for the UI elements that lie below. Such automaticscaling and positioning may be determined and saved per application, andin some cases per context within an application. More specifically, theMT pointer is set to a size whereby the digit indicators of the MTpointer are sized and distanced similarly to the size and distance of UIelements on-screen at the current position of the MT pointer. However,different UI elements may differ in size and distance. Accordingly, someembodiments of the MTI provide a touch gesture for customizing theoverall size of the MT pointer and, as a result, the size of theconstituent digit indicators, so that the user can override anyautomatic setting. Some embodiments of the MTI provide a different touchgesture for customizing the distance between and relative position ofthe MT pointer digit indicators.

In some embodiments, a touch gesture invokes a size adjustment functionto adjust the size of the MT pointer and, thereby, adjust the size ofthe constituent digit indicators. This function is useful to adjust theoverall size of the MT pointer so that the MT pointer digit indicatorscan be more closely aligned to the UI elements based on the size andspacing of the UI elements on-screen. In some embodiments, to invoke thesize adjustment function, the user performs a simultaneous four fingertouch on the trackpad. Once invoked, the user can reduce the size of theMT pointer and its digit indicators by sliding the four touches down.Conversely, the user can increase the size of the MT pointer and itsdigit indicators by sliding the four touches up.

FIG. 8 illustrates performing a four touch slide gesture to increase thesize of the MT pointer and its digit indicators in accordance with someembodiments. The figure illustrates a trackpad 810 on which the fourtouch upward slide gesture is performed. Each dotted “X” represents aninitial touch position of a finger of the right hand and each solid “X”represents the final position of a finger of the right hand after thegesture is complete. The figure also shows how this gesture increasesthe size of the MT pointer by illustrating the size of the MT pointerbased on the initial MT pointer 820 and the size of the MT pointer 830after the four touch slide up gesture is performed on the trackpad 810.

To implement the size adjustment function, the MTI intercepts orotherwise obtains touch data generated by the trackpad based on usertouches. From the touch events, the MTI identifies four separate butessentially simultaneous touches where the touches correspond to somedegree with known locations for the index, middle, ring and littlefinger touches and spacing. Then when the four touches are slid up ordown, the MTI tracks the change or delta in the positions of the foursets of coordinates. From these deltas, the MTI can compute whether aresize gesture occurred, and if so, change the size of the MT pointerand the digit indicators in proportion to an average delta value.Relative spacing of the digit indicators is changed proportionally, butinter-digit spacing ratios remain constant. It should be well noted thatthe foregoing gesture is given as an example, and other gestures and/orkeys (as described in the following section C) may be employed to modifythe size of the MT pointer.

In contrast to modifying the size of the MT pointer, some embodimentsalso provide a digit indicator position adjustment function that can beused to independently adjust the position of each MTI digit indicator ofthe MT pointer. This function is useful to adjust the position of the MTpointer digit indicators so that they are better positioned inorientation and distance with the position of the UI elements on-screen.For example, UI elements of an application are often horizontallyaligned. Therefore, by adjusting the position of the MT pointer digitindicators so that they too are horizontally aligned, the digitindicators can be used to more easily access the horizontally aligned UIelements without re-positioning the MT pointer as a whole. In someembodiments, the digit indicator position adjustment function is invokedwhen the user holds two or more touches, and while retaining thosetouches on the trackpad, moves of any one or more touches produce apositional adjustment to the corresponding digit indicators of the MTpointer.

FIG. 9 illustrates performing a four finger touch and subsequentlymoving the touch of only the index finger to adjust the relativeposition of the index finger digit indicator of the MT pointer inaccordance with some embodiments. The figure illustrates a trackpad 910on which the four finger touch is performed. The dotted “X” representsan initial touch position of the index finger and each solid “X”represents the final position of a finger of the right hand, which, inthe case of the non-moving middle, ring, and little fingers is also theinitial position. The figure shows how moving the index finger touchfrom the initial position to its final position changes the positioningof the index finger digit indicator of the MT pointer from an initialdigit indicator position configuration 920 to a final digit indicatorposition configuration 930.

Different embodiments of the MTI utilize different touch combinations,gestures, and movements to perform the above described size adjustmentfunction and the digit indicator position adjustment function.Accordingly, it is not necessary to implement these functions based on afour finger touch. For example, with reference to FIG. 9, someembodiments of the MTI do not require the indicated touches of the ringand little fingers to invoke the described functionality. As anotherexample, FIG. 10 illustrates performing a five finger expansion gestureto reposition each of the digit indicators of the MT pointer. As shown,each of the user's fingers are moved out and away from one another onthe trackpad 1010 and, in response, the MT pointer digit indicatorschange from an initial position 1020 to a final position 1030. It shouldbe noted that the relative offsets that are used in the mathematicalmodel of the MT pointer are altered in the digit indicator positionadjustment function, unlike the size adjustment function where theoffset ratios remain consistent.

C. MTI Operation with Keyboard

In some embodiments, the MTI is controllable using a traditionalkeyboard independent of or in conjunction with a mouse, trackpad,touchscreen or other similar spatially dependent input device.

a. Traditional Keyboard with Pointing Device

In this method of MTI operation, movement of the MT pointer iscontrolled with a mouse, trackpad, or other input device, and keyboardkeys activate the MTI digit indicators. In some embodiments, thekeyboard home keys activate the MTI digit indicators. Some embodimentsallow the user to use the left hand or thumb to operate a trackpad orsimilar device to move the MT pointer and the fingers of the right handto press (1) the “J” keyboard key to activate the index finger digitindicator of the MT pointer, (2) the “K” keyboard key to activate themiddle finger digit indicator of the MT pointer, (3) the “L” keyboardkey to activate the ring finger digit indicator of the MT pointer, and(4) the “Semicolon” keyboard key to activate the little finger digitindicator of the MT pointer. Other embodiments allow the user to use ofthe right hand or thumb to operate a trackpad or similar device to movethe MT pointer and the fingers of the left hand to press (1) the “F”keyboard key to activate the index finger digit indicator of the MTpointer, (2) the “D” keyboard key to activate the middle finger digitindicator of the MT pointer, (3) the “S” keyboard key to activate thering finger digit indicator of the MT pointer, and (4) the “A” keyboardkey to activate the little finger digit indicator of the MT pointer. Inother embodiments, other keys or key combinations are used.

The above enumerated home keys may be used to perform mouse right-clickactivation with a corresponding digit indicator of the MT pointer.Right-click activation may also be provided by using a row of keyboardkeys above or below the home keys. For example, when using the righthand or thumb to operate a trackpad or similar device to move the MTpointer, the “R” keyboard key can be used to perform mouse right-clickactivation with the index finger digit indicator of the MT pointer, the“E” keyboard key can be used to perform mouse right-click activationwith the middle finger digit indicator of the MT pointer, the “W”keyboard key can be used to perform mouse right-click activation withthe ring finger digit indicator of the MT pointer, and the “Q” keyboardkey can be used to perform mouse right-click activation with the littlefinger digit indicator of the MT pointer.

To integrate MTI functionality with traditional keyboard usage (i.e.,text insertion), some embodiments utilize a particular keyboard key,such as the spacebar, to enable MTI functionality. The spacebar is thepreferred keyboard key to enable the MTI functionality in someembodiments, because it is easily accessible with the thumb of eitherthe left hand or the right hand and it allows the other fingers toremain positioned on the home keys. When the spacebar is not pressed,the MTI functionality is disabled in some embodiments. When disabled,the GUI pointer is a traditional single pointing tool. When the spacebaris pressed and held down beyond a specified threshold duration (e.g.,200 milliseconds), the MTI functionality is enabled and the GUI pointerchanges to the MT pointer. By continuing to hold down the spacebar, thehome keyboard keys can be used to activate different MT pointer digitindicators. For instance, holding the spacebar down and tapping the “J”key activates the index finger digit indicator of the MT pointer wherebythe activation simulates performing a mouse left-click on the UI elementover which the index finger digit indicator of the MT pointer ispositioned. To implement this and similar methods of operation, the MTIintercepts or otherwise obtains data and/or events generated as inputsfrom the keyboard and other input devices. Such data and/or events arethen processed to determine whether to enable MTI functionality. Whendisabled, the inputs are passed through to the OS or application eventhandler. When enabled, the inputs are processed to determine whether toactivate the MTI digit indicators, move the MT pointer, or provide otherfunctionality.

b. Traditional Keyboard without Pointing Device

MTI functionality can be provided so that a traditional keyboard can beused to fully utilize the MTI without the need of a separate inputdevice such as a mouse or trackpad. In some such embodiments, holdingdown the spacebar enables MTI functionality whereby home keys associatedwith a first hand (e.g., left or right hand) are usable to activate MTpointer digit indicators and home keys associated with a second hand areusable to perform MT pointer movements wherein the home keys associatedwith the second hand can be used to perform different directionalon-screen incremental movements of MT pointer. More specifically, thehome keys associated with the second hand are adapted to perform arrowkey functionality for moving the MT pointer up, down, left, right, ordiagonally. In this method of MTI operation, the user's hands remain onthe keyboard when performing traditional text insertion functionalityand when performing MT pointer movements and MT pointer digit indicatoractivation. To further facilitate MT pointer movement, some embodimentscontinue to move the MT pointer while certain keys remain depressed, andMT pointer movement is stopped when certain keys are released. Suchmovement may accelerate as the key remains depressed. In someembodiments, pressing additional keys while the cursor is in motionshifts the direction of movement. Pressing other additional keys mayspeed up the rate at which the MT pointer moves across the screen.

For example, FIG. 11 illustrates an embodiment whereby a standardkeyboard 1110 is used to operate the MT pointer, including moving the MTpointer on the screen. Pressing and holding the space bar 1120 causesthe MT pointer 1130 to appear on-screen. In some embodiments, ananimation is presented which helps bring the user's focus to thelocation of the MT pointer 1130. Also, an on-screen keyboard 1140appears which identifies the various keys of the keyboard 1110 that maybe utilized to invoke various MT pointer 1130 functionality or otherfunctionality. As shown, the “F” key can be used to move the MT pointer1130 right, the “D” key can be used to move the MT pointer 1130 down,the “S” key can be used to move the MT pointer 1130 left, the “E” keycan be used to move the MT pointer 1130 up, the “J” key can be used toactivate the index finger digit indicator, the “K” key can be used toactivate the middle finger digit indicator, the “L” key can be used toactivate the ring finger digit indicator, and the “Semicolon” key can beused to activate the little finger digit indicator. Accordingly, whenthe “F” key is pressed, the MT pointer 1130 moves to the right.

In some embodiments, pressing or holding any of the “E”, “S”, “D”, and“F” keys also changes the on-screen keyboard 1140 to show a new ormodified set of options. For example, pressing and/or holding the “F”key with the “D” key (down) causes the MT pointer 1130 to move in boththe leftward and the downward directions at a 45 degree angle until the“D” key is released and then movement continues to the right until the“F” key is released. As another example, when pressing or holding any ofthe “E”, “S”, “D”, and “F” keys, the “L” key can also be pressed toincrease the speed with which the MT pointer 1130 moves in the directionassociated with the “E”, “S”, “D”, or “F” key being pressed or held.Releasing the “L” key resumes the normal slower movement speed.

In some embodiments, an on-screen indicator, such as an arrow, appearson-screen to clearly show the current movement direction and to aid theeye in tracking the MT pointer 1130 as it moves across the screen. Withthe MT pointer 1130 properly positioned on a desired UI element, theuser taps the “J” key to activate the UI element below the index fingerdigit indicator of the MT pointer 1130. Finally, the user releases thespace bar 1120 which hides the MT pointer 1130 and the on-screenkeyboard 1140, and the user may resume normal typing.

c. Enhanced Keyboard with Single Touch Key

In some embodiments, the MTI is operable using an enhanced keyboard thatincludes at least one touch sensitive key and software for adaptingtouches applied over the touch sensitive key or keys to emulate trackpadfunctionality. The software may be stored on specialized hardware of theenhanced keyboard, such as in a controller of the enhanced keyboard. Insome such embodiments, the controller is implemented with firmware thatis executed using an integrated circuit (IC) of the enhanced keyboard.The controller converts raw touch data sensed over the touch sensitivekey and keyboard key presses and key releases into appropriate real-timemessages that are then communicated via an appropriate protocol such asUSB or Bluetooth to the host device (i.e., the OS of the computingsystem) on which the MTI executes.

In a preferred single touch sensitive key embodiment shown in FIG. 12,the touch sensitive key is the spacebar key 1210. In some embodiments,the touch sensitive spacebar 1210 is enlarged with respect to atraditional spacebar key in order to provide a larger touch surface overwhich trackpad functionality can be emulated. The touch sensitivespacebar key 1210 includes one or more touch sensitive layers thatprovide touch and motion detection over the key using capacitive,resistive, or other touch or presence sensing technologies. When theuser's fingers are over the home position keyboard keys, the naturalresting position of the thumb of either hand is over the spacebar 1210.The thumb of either hand can therefore be used to perform varioustouches or movements over the touch sensitive spacebar 1210 withoutremoving the other fingers from atop the home keyboard keys (e.g., “A”,“S”, “D”, “F”, “J”, “K”, “L”, and “Semicolon” keys).

In some embodiments, touch movements performed on the spacebar 1210 whenthe spacebar 1210 is not fully pressed down can be used to performtraditional pointer movements onscreen. In some embodiments, holdingdown the spacebar 1210 enables the MTI functionality. Then whilecontinuing to hold down the spacebar 1210, touch movements that areperformed on the touch sensitive spacebar 1210 may be used to move theMT pointer as shown in FIG. 13. When the MTI functionality is enabled,the home keyboard keys are usable to activate the MT pointer digitindicators as shown in FIG. 14. Specifically, FIG. 14 illustrates thecorrespondence between the home keys of the enhanced keyboard and thedigit indicators of the MT pointer in accordance with some embodiments.When the MTI functionality is enabled, the “J” keyboard key 1410 can beused to activate the index finger digit indicator 1420, the “K” keyboardkey 1430 can be used to activate the middle finger digit indicator 1440,the “L” keyboard key 1450 can be used to activate the ring finger digitindicator 1460, and the “Semicolon” keyboard key 1470 can be used toactivate the little finger digit indicator 1480. In other embodiments,the MTI functionality is enabled by pressing and holding a differentkeyboard key or keys, or by a gesture or other sensing means.

Other embodiments of the enhanced keyboard are shown in FIGS. 15 and 16.FIG. 15 illustrates adapting the touch surface of the touch sensitivespacebar keyboard key to provide different functionality depending onwhere a touch or tap is applied to the spacebar. Specifically, in someembodiments, the touch surface of the spacebar is split into two regions1510 and 1520. Taps applied over the first region 1510 may beinterpreted by the enhanced keyboard controller and associated softwaredifferently than taps applied over the second region 1520. For example,in some embodiments, a tap to the first region 1510 is interpreted as amouse left-click without having to hold down the spacebar and a tap tothe second region 1520 is registered as a mouse right-click only whenthe spacebar is held down. This region sensing capability is combinedwith previously disclosed thumb/finger contact area sensing for moreaccuracy in some embodiments. Such region sensing areas include two ormore non-rectangular areas in some embodiments, and such areas maychange dynamically in response to user activity and/or system state. Forexample, in some embodiments the area where the user normally rests theleft thumb may be tracked, and subsequent touches and taps to that areaare not translated into a mouse click or any other message, as thiscontact area is used to depress and hold the spacebar down for MTIactivities, and/or enable scrolling in conjunction with slides on otherareas or keys, or as a modifier key used in combination with flag keys.

In some embodiments, the touch surface of the spacebar of the enhancedkeyboard recognizes multiple touches as distinct touches. As such,multi-touch gestures may be performed on the spacebar to emulatemulti-touch trackpad functionality. Moreover, the digit indicators ofthe MT pointer can be activated directly using the touch surface of thespacebar. FIG. 16 illustrates usage of the enhanced keyboard withfingers of one hand providing touches and gestures on the spacebar wheresuch touches and gestures can be used to emulate traditional trackpadfunctionality, move the MT pointer when the MTI is enabled, and activatethe various digit indicators of the MT pointer when the MTI is enabled.The use of gestures on the spacebar may be enabled with the spacebar inthe up or down state, and as previously mentioned, a thumb resting orpressing the spacebar may be essentially filtered so that the touchevent(s) do not affect the gesture(s) performed on other areas of thespacebar. Further description of the enhanced keyboard is provided inUnited States Provisional patent application entitled “Input Device UserInterface Enhancements”, having Ser. No. 61/465,345, and a filing dateof Mar. 17, 2011.

D. MTI Implementation

In some embodiments, the MTI is implemented as a software application ora background process of an operating system (OS). In some otherembodiments, the MTI is integrated as part of operating system (OS) orGUI functionality. Through the use of various interfaces, system calls,application programming interface (API) calls, or other integrationtools, the MTI gains access to data and events that are provided byinput devices such as a mouse, trackpad, keyboard, enhanced keyboard,etc. It should be apparent that different operating systems andelectronic devices provide different means for accessing such data andevents. Accordingly, different embodiments of the MTI are adapted toobtain the data and events using the appropriate interface or calls. TheMTI processes the data and events to provide the above described MTIfunctionality.

a. MT Pointer Movement

Core MTI functionality involves tracking MT pointer movement as a wholeand activating the appropriate individual MTI digit indicators. FIG. 17Aconceptually illustrates an internal mathematical model for tracking anon-screen position of the MT pointer in accordance with someembodiments. In this implementation, the MT pointer is represented as acenter point 1710 with offsets 1720, 1730, 1740, 1750 and 1760. Thecenter point tracks the current on-screen position of the MT pointer.Each offset 1720-1760 represents a position of a digit indicatorrelative to the center point 1710. In some embodiments, the center point1710 is an x-y on-screen coordinate and each offset is an x-y deltavalue identifying a directional distance from the center point 1710. Thegraphical representation for the MT pointer is approximately centeredabout the center point 1710 when displayed on-screen and the graphicalrepresentation for each digit indicator is shifted to extend from thecenter point 1710 by the corresponding offset value, and at a rotationthat emulates human hand geometry for either the left or right hand. Insome embodiments where touch shape and angle data are available, therotation value is derived from the actual angle of the touch. This isespecially important for the thumb digit display, as left-right thumbmovement should also affect the rotation value to appear realisticon-screen.

When the MT pointer is moved using a keyboard, mouse, trackpad, touchsensitive spacebar, or other pointing device, the MTI intercepts thedata and events that quantify the movement, and the MTI adjusts theposition of the center point 1710 relative to intercepted data orevents. In embodiments where a trackpad or other multiply touchsensitive device is used with the MTI, a single touch with subsequentslides is used to make corresponding movements of the MT pointer. Two ormore essentially simultaneous touches which are then slid in anessentially fixed-relationship fashion also cause a correspondingmovement of the MT pointer as a whole. Two or more touches that are heldand then slid in essentially non-fixed-relationship fashion cause theinter-digit indicator offsets to change so that they reflect the offsetsof the fingers on the trackpad. In some embodiments, when the MT pointeris active, a highlight ring appears for touches that are currently incontact with the trackpad.

In some embodiments, the position of the MT pointer tracks the existingGUI pointer of the operating system, such that the x-y coordinates ofthe GUI pointer are equal to the center of the index finger digitindicator. Offsets to the other digit indicators and center point remainconsistent. When the MTI is invoked, the GUI single pointer is hiddenand the MT pointer appears such that the position of the index fingercenter is in its place. In other embodiments, the position of the MTpointer is independent of the GUI pointer, and the last position of theMT pointer is used as the starting point when the MTI is newly invoked.In some embodiments, the position of the MT pointer is saved on aper-application basis so that the user may leave the MTI positionedabove certain UI elements in each application. This per-application MTpointer data is saved so that when the application is re-launched thepersisted MT pointer position is restored. When the MT pointer is firstinvoked in an application which has no such saved location data, the GUIpointer is used as the index finger digit indicator location.Furthermore, some embodiments provide for various locations of the MTpointer and digit indicators to be saved, along with the UI elementsthat have been invoked at that proximate position. This allowsfunctionality such as the MT pointer position snapping to previous knownlocations, and also requiring confirmation in some cases when a UIelement has not been previously activated by a particular digitindicator.

As described in the operation section above, the size adjustmentfunction and the digit indicator position adjustment function adjust theoffsets 1720-1760 relative to the center point 1710. In someembodiments, the size adjustment function is performed by applying amultiplier to the offsets 1720-1760, where the multiplier is greaterthan one when increasing the size of the MT pointer and less than onewhen decreasing the size of the MT pointer. The digit indicator positionadjustment function is used to individually adjust the relativepositions of each of the digit indicators. When the digit indicatorposition adjustment function is performed for a particular digitindicator, the offset for a particular digit indicator is adjustedrelative to the center point 1710. In other embodiments, the offsets aredetermined directly from touch data that corresponds with each digit.

FIG. 17B illustrates MT pointer movement in an embodiment whichprocesses touch data from a device such as a trackpad which tracksmultiple digits. An MT pointer with all 5 digits touching 1700 b isindicated by dark circles centered at each of the touch locations. Thesame MT pointer when the 5 touches are removed 1705 b is distinguishedby dotted circles. When the thumb is touching 1710 b and is then slid toa new location 1715 b, the MT pointer maintains the inter-digitrelationships such that the entire MT pointer tracks the thumb movement.A touching index finger 1720 b can also be used to move the entire MTpointer 1725 b to various on-screen locations. In some embodiments, whenany single touching digit slides or flicks on the touch surface, theentire MT pointer moves as a single entity, responding in a wayanalogous to traditional single pointer movement, with settings andalgorithms for tracking speed, simulated inertia, etc. When two fingersare touching 1730 b and slid to a new location 1735 b in an essentiallyparallel or fixed-relationship manner, the MT pointer also moves as asingle entity to the new location. When two touches 1740 b are moved ina non-parallel or non-fixed-relationship manner 1745 b, the MT pointercenter point is moved in relation to the touching digit movements, withoffsets updated to that new center point, and non-touching digits aremoved according to their pre-existing offsets. When the touches arelifted, the inter-digit spacing, or shape, of the MT pointer isretained. The ability to change the inter-digit spacing allows the userto change the shape of the MT pointer easily. When a touch of the littlefinger is applied 1750 b and then slid to a new location 1755 b, the MTpointer is moved as a group and the shape of the MT pointer is retained.To more closely emulate human hand movement geometry, some embodimentscalculate thumb movements independently from finger movements. Forexample, when a thumb touch is added to the previous little finger touch1760 b, a slide of the thumb (while the little finger remainsstationary) 1765 b causes the thumb to move (and change angle) while allfingers remain stationary. This is in contrast to a thumb movement whereno fingers are touching 1715 b, which moves the entire MT pointer. Ifany finger is touching, thumb movement is treated independently, and insome embodiments is offset from the index finger rather than the centerpoint. From the previous position 1770 b, when all digits touch 1775 b,the MT pointer assumes a new shape by updating the inter-digit spacingto reflect the current touch positions. Thus, the MT pointer moves bothas a group based on overall digit movement, and independently based oninter-digit relationships within the overall movement.

In some embodiments, the MT pointer is assigned a default size andposition for the digit indicators. When either the size of the MTpointer or the position of the digit indicators is changed, the MTIstores the change so that any changes are retained whenever the MTI isenabled. Some embodiments store the size of the MT pointer and theposition of the digit indicators on a per application basis. This allowsfor the MT pointer to be customized according to the layouts ofdifferent applications whereby a first application may have UI elementssized and spaced differently than a second application. By providing theper application customizations, users can adjust the size of the MTpointer and offsets of the digit indicators once per application andwhen that application becomes frontmost, the MT pointer and digitindicators change according to the settings specified for thatapplication. To provide such customizations, the MTI includes a databaseor a configuration file to store the per application customizations.Additionally, the MTI tracks the frontmost application using systemstate information that is obtainable via various system or applicationprogramming interface (API) calls. In some embodiments, the size and orspacing of the MT pointer changes depending on the position of the MTpointer. Specifically, the UI elements below the MT pointer adjust theposition of individual digit indicators in order to match positions asclosely as possible for a one-to-one correspondence between the digitindicators and the UI elements below.

b. MTI Digit Indicator Activation

To activate the appropriate digit indicator when the MTI is touchoperated (using a trackpad or touch sensitive enhanced keyboard), theMTI performs one or more touch assignment processes. The MTI includeslogic for determining which one or more fingers of which hand are usedto perform each touch. The logic performs a touch assignment wherebyeach detected touch is assigned to control operations of a particularfinger (e.g., index, middle, ring, or little). In some embodiments, thetouch assignment is performed relative to home positions that areestablished from a previously issued four finger touch, or a five fingertouch if the thumb is touching.

i. Establishing Home Positions

In some embodiments, the MTI establishes a home position for each of theindex, middle, ring, and little fingers whenever four essentiallysimultaneous touches are registered as a valid four finger touch. Thehome positions are retained in memory until a new valid four essentiallysimultaneous touches are performed. As will become evident from thediscussion below, the home positions are used in determining whichfinger is assigned to each touch. As noted above, a touch of the thumbof either the left or right hand is distinguishable from touches ofother fingers because of the geometry (e.g., size, shape, and angle) ofthe contact area of a touch produced by the thumb.

Upon receiving a valid four finger touch from which home positions canbe established, the MTI determines (i) a center point of the four fingertouch and (ii) offsets of each touch from the center point. The centerpoint is a single point of a mathematical model which tracks movement ofthe user's hand in order to keep the home positions aligned with theuser's hand as new touches are added and existing touches are moved. TheMTI derives the home positions based on the offsets from the currentcenter point position. Each of four offsets from the current centerpoint position establishes a home position for a particular touch of theindex, middle, ring, and little fingers. Accordingly, the center pointis a single point from which individual digit proximities can be derivedwhen analyzing new touches, and by which existing touches that move canbias the center point so it can be used as a basis for futurecalculations. Once the offsets for each touch are computed, themathematical model is established and predictive zones for each homeposition are calculated.

ii. Proximity Assignment

Proximity assignment involves comparing the x-y coordinates of adetected touch to the x-y coordinates of previously established homepositions. If the touch is a tap gesture, the detected touch is thenused to activate the MT pointer digit indicator that is associated withwhichever finger's home position is closest to the detected touch.

For a single finger tap, some embodiments identify which digit indicatorof the MT pointer to activate by comparing the x-y coordinates of thetap to the x-y coordinates of established home positions. The tapactivates the digit indicator with established home position x-ycoordinates that are closest to the x-y coordinates of the tap. Otherembodiments utilize more complex algorithms wherein home positions maybe biased by previous activity such as slides. Still other embodimentstake into account information about the UI element that would be thetarget of the proposed tap. For example, an assignment to the middlefinger using a basic algorithm may not yield a valid UI element, whereasan assignment to the index finger would yield a valid UI element.Furthermore, historical data representing previous successful UI elementinvocations with the proposed finger assignment are included in theanalysis of some embodiments.

For a two or more finger touch or tap, the digit indicatoridentification/assignment is slightly more complex. Since two differentdetected touches can both be closest to a single home position, the MTIperforms the proximity assignment process in two passes as describedbelow in FIG. 18A. FIG. 18A presents a proximity assignment process 1800a performed by the MTI to determine which touch corresponds with whichfinger in accordance with some embodiments. The process begins when oneor more touches are newly detected.

The process identifies (at 1810 a) the x-y coordinates for the one ormore new touches. The process performs (at 1820 a) a first proximitycomputation pass. For the first proximity computation pass, the process(i) identifies which previously established home position is closest tothe left of each newly detected touch and (ii) computes a delta ordistance between the x-y coordinate for each newly detected touch andthe x-y coordinate for the home position that is closest to the left ofthat newly detected touch. The process sums each of the deltas computedat step 1820 a to compute (at 1830 a) a first total delta value. Theprocess then performs (at 1840 a) a second proximity computation pass.For the second proximity computation pass, the process (i) identifieswhich previously established home position is closest to the right ofeach newly detected touch and (ii) computes a delta between the x-ycoordinate for each newly detected touch and the x-y coordinate for thehome position that is closest to the right of that newly detected touch.The process computes (at 1850 a) a second total delta value by summingeach of the deltas computed at step 1840 a.

The process compares (at 1860 a) the first total delta value to thesecond total delta value to identify the lowest total delta value. Thiscomparison determines how the MTI will assign the newly detected touchesand, as a result, which digit indicator will be activated by the newlydetected touch. When the first total delta value is less than the secondtotal delta value, the process assigns (at 1870 a) each new touch toperform operations that are associated with the finger corresponding tothe home position that is closest to the left of that new touch.Otherwise, the process assigns (at 1880 a) each new touch to performoperations that are associated with the finger corresponding to the homeposition that is closest to the right of that new touch. Other touchassignment processes (e.g., shift assignment and release assignment) aredescribed in the United State Provisional patent application entitled“Touch Enhanced Interface”, having Ser. No. 61/465,309, and a filingdate of Mar. 17, 2011.

iii. Shift Assignment

Shift assignment is used in some embodiments when home positions are notsuitable, such as when the user has repositioned the hand after someperiod of time, or after switching back from keyboard operation. FIG.18B provides several examples of shift assignments which map touches tofingers without regard to previous touch assignment locations. When asingle touch is detected as depicted in 1811 b as a bold circle abovethe MT pointer, the touch is mapped to the index finger as indicated bythe bold circle on the index finger in 1811 b. If two touches areessentially simultaneously detected 1812 b, they are assigned to theindex and middle fingers if the x-axis spacing is approximately fingerwidth apart. If two touches are wider apart, index and ring assignmentis determined, and even wider two touch spacing 1814 b results in anindex, little finger assignment. Some embodiments track finger widthspacing dynamically and per user. If three essentially simultaneoustouches are received 1821 b, they are assigned to the index, middle andring fingers unless there is a significant x-axis space between two ofthe digits. If the significant space is between the second and thirdtouches 1822 b, index, middle and little finger assignments wouldresult. Subsequently, if a new received touch has an x-axis coordinatethat is between the x-axis of the middle and little finger 1823 b, thenew touch is assigned to the ring finger. As depicted for 1823 b and thesubsequent diagrams, dotted circles represent previous touches thatremain held down and bolded solid circles represent new touches. Unlike1823 b, if the new touch has an x-axis coordinate that is greater thanthe current little finger assignment's x-axis position 1824 b, thisindicates that a re-assignment must occur such that the touch of thelittle finger is re-assigned to the ring finger, and the new touch isassigned to the little finger. As another example of re-assignment, ifthree fingers are assigned to the index, middle and ring fingers 1821 b,and a new touch is detected with an x-axis coordinate that is to theleft of the other touches 1831 b, re-assignments must occur toaccommodate the new touch assignment to the index finger as indicated bythe dotted line arrows. Similarly, if a single touch is assigned to theindex finger and a new touch has a lower x-axis value 1832 b, the touchof the index finger is re-assigned to the middle finger to accommodatethe assignment of the new touch to the index finger. If there issignificant x-axis spacing between the previously assigned touch and thenew touch 1833 b, the previous touch will be re-assigned to the ringfinger, and the new touch assigned to the index finger. If fouressentially simultaneous touches are received 1834 b, the assignment isdetermined by the x-axis ordering of the touches. In practice, if theuser wishes to assign a touch to, for instance, the little finger, theycan simply touch all four fingers down and then release the index,middle and ring fingers.

It should be noted that touches may begin, move and end at any time andin any combination, so the MTI system must accommodate more scenariosthan the representative examples just given. Furthermore, thumbdetection is also considered in assignment and re-assignment, based onthe geometry and attributes of received touches. FIG. 18C provides aprocess used in some embodiments to make assignment determinations whenfinger, thumb and resting touches are considered. The process 1800 cstarts when a new touch is received (at 1805 c). The MTI system maycoalesce multiple touches that began essentially simultaneously so thatthey can be evaluated as a group, which aids in the assignment process.At 1810 c a determination is made about the actual contact shape of thetouch or touches. Typically, the contact area for a finger is rounder(less elongated). This and other attributes, including user touchhistory are used to determine if the touch or touches were generated bya finger as opposed to a thumb or a resting palm. A finger designation(at 1820 c) expects one to four touches corresponding with the fingersof either left or right hand. The left-right hand determination is madein some embodiments by touching all 5 digits, at which time the handorientation is set until it is changed by the user. For clarity, thebalance of this process will assume right hand orientation. At 1835 c adetermination is made as to whether any previously assigned fingers arestill touching. If not, assignments are made according to the number ofessentially simultaneous touches and their positions. If there is onlyone detected touch, it is assigned (at 1840 c) to the index finger. Ifthere are two touches, the leftmost touch is assigned (at 1845 c) to theindex finger and the rightmost to the middle finger. FIG. 18B providessome detail about the spacing of the touches as to whether the rightmosttouch is potentially mapped to the ring or little finger. For the sakeof clarity, fingers in this process are assumed to be not broadlyspaced. If three fingers are detected, they are assigned (at 1850 c) tothe index, middle and ring fingers, and if four fingers are detected,they are assigned (at 1855 c) by increasing x-axis position to theindex, middle, ring and little fingers. If five fingers are detected,this is an indication that a previous finger/thumb determination wasincorrect. Further analysis is utilized to re-assign (at 1860 c) one ofthe fingers as a thumb. Such analysis usually concludes that the lowesty-axis touch, and usually the lowest x-axis touch should be assigned tothe thumb. If it was determined (at 1835 c) that there were pre-existingfinger assigned touches, the process determines the x-axis ordering ofall finger touches and either fits (at 1870 c) the new touches insequentially with the prior finger touches, or determines that certainof the prior finger touches must be re-assigned. Further detail on thisaspect of the process is provided in FIG. 18B above. If the initialshape and attribute determination (at 1810 c) determines that a touchwas not made by a finger, the process next determines (at 1815 c) if thetouch is at the bottom edge of the device or exhibits othercharacteristics of a touch that is resting on the surface. If sodesignated, the touch is assigned (at 1825 c) as a resting touch and itdoes not figure into further hand geometry calculations nor trigger anyoutput action. If the determination (at 1815 c) is not a resting touch,then it is determined (at 1830 c) to be a thumb touch. The processchecks (at 1875 c) to see if there is already an existingthumb-designated touch down at, and if not the touch is assigned (at1880 c) to the thumb. If there were more than one new touches that werecharacterized as thumb touches, or if there was an existing thumb touchdown when a new one was detected, the process re-evaluates (at 1885 c)the touches based on the current touch data and locations to re-assign(at 1820 c) some touches as finger touches.

iv. Hybrid Assignment

The MTI system employs various digit assignment processes such as theproximity and shift assignment processes detailed above. In someembodiments, combinational assignments are utilized in series or intandem to improve the accuracy of touch to digit assignments.Furthermore, the MTI can accept input from a variety of presence sensingand other technologies (such as image or infrared sensors) to improvethe accuracy of the touch to digit assignment process. For example, atrackpad or other sensing hardware can be used by the MTI to monitor auser's hand, finger, and palm positions even when no touches aredirectly applied to the touch surface of the trackpad or other sensinghardware. Specifically, the sensing capabilities of the trackpad orother sensing hardware can be enhanced to identify a user's handproximity as it hovers over the touch surface. It should be noted thatthe accuracy of this proximity does not necessarily need to be precisein order to still provide cues to the MTI, so that it can increase theaccuracy of the touch to digit assignment process.

MTI digit indicator activation is simpler process when a keyboard isused as a digit indicator activation mechanism. Specifically, when theMTI is enabled, a “J” keyboard key press activates the index fingerdigit indicator, a “K” keyboard key press activates the middle fingerdigit indicator, an “L” keyboard key press activates the ring fingerdigit indicator, and a “Semicolon” keyboard key press activates thelittle finger digit indicator.

Digit indicator activation involves issuing an event that represents theactivation at the current location of the corresponding digit indicator.The event may include a mouse left-click action, a mouse right-clickaction, a mouse double-click action, a hold action, a drag action, ascroll action, etc. The specified location is determined based onsumming the center point of MT pointer with the offset position of theactivated digit indicator. The event is passed to the OS or theappropriate frontmost application for processing. In some embodiments,the MTI is unconcerned with whether activation of a digit indicatorresults in any functionality being performed (e.g., selection being madeor output action being invoked). The determination as to whether theissued event produces any functionality is left to the OS or frontmostapplication event handler.

c. MTI UI Element Disambiguation

In a traditional GUI pointer, there is an attribute called a “hotspot”which is the exact single x-y coordinate at which UI elementdisambiguation occurs. For example, the arrow pointer in many existingGUI implementations is an icon that looks like an arrow pointing up andto the left. If only the tail end of this icon is placed above a UIelement and the user clicks, activation of that UI element will notoccur, because only the hotspot of the arrow icon is relevant, and therest of the icon is as a visual aid only. The position of the hotspotmay change depending on which pointer icon is current. In the case ofthe above described pointer icon, the hotspot is typically near the topleft of the arrow icon.

The present invention departs from this single-coordinate hotspotparadigm in favor of a “hotzone”, wherein there is a much larger area inwhich the user may click on or otherwise access a particular UI element.This allows the user to more easily navigate the UI, because lessprecise pointer movement is required. In some embodiments, the hotzoneis displayed on-screen as a transparent circular area at the end of asemi-transparent digit indicator.

FIG. 19 illustrates hotzones 1910, 1920, 1930, 1940, and 1950 that areassociated with each digit indicator of the MT pointer 1960 inaccordance with some embodiments. As shown, the hotzones 1910-1950include circular regions toward the end of each digit indicator. Itshould be apparent that a hotzone can be any shape comprising an areainstead of a single point and need not be embodied as a circular region.Furthermore, it is not necessary for each digit indicator to beassociated with a hotzone. For example, the hotzone 1910 for the thumbdigit indicator may be omitted in some embodiments.

When the user moves the MT pointer 1960, one or more digit indicatorsand their hotzones analyze the UI elements below in order to determinewhich particular UI element (if any) should be highlighted, selected,activated, or otherwise manipulated or utilized. This UI elementdisambiguation for a particular hotzone is streamlined if there are noother UI elements within the hotzone's area. A further analysis isrequired if there are two or more UI elements within the hotzone. SuchUI element disambiguation is described with reference to FIGS. 21-25below.

In a traditional GUI, there are many types of UI elements such aswindows, text boxes, scrolling fields, sliders, buttons, etc., and theseUI elements can be superimposed atop one another in a hierarchy so that,for instance, a button is always atop the window which contains it. Thisallows the user to click on the button if the pointer is within thebounds of the button, or on the window if the pointer is outside thebounds of the button but within the bounds of the window. In this case,the window may be called a “container” UI element, since it containsother UI elements. Furthermore, clicking on a non-button area of thewindow may not trigger any particular action. The MTI analyzes this UIelement hierarchy and other attributes to determine which UI elementsthe hotzone should consider in the disambiguation process.

FIG. 20 presents the disambiguation process 2000 that is performed bythe MTI in accordance with some embodiments. The process begins byobtaining (at 2010) a list of current on-screen UI elements and theircorresponding boundary coordinates, containment, and other attributes.This list may be obtained from the OS or other system software through aseries of API or system calls.

For each UI element in the obtained list that is relevant, the processdetermines (at 2020) an inset region and an outset region. To determinethe inset region and the outset region, the process first computes theradius of the digit finger hotzone. As the hotzone region for each digitfinger indicator has the same area, the radius is a value that is sharedby all digit indicators. Next, the process determines the inset regionfor a UI element by shrinking the boundary for the UI element by theradius. The process determines the outset region for the UI element byexpanding the boundary for the UI element by the radius. The inset andoutset regions can be any convex shape, however for purposes of clarity,rectangular inset and outset regions are shown in the figures below.FIG. 21 illustrates an inset region 2110 and an outset region 2120 for arectangular shaped UI element 2130. The original boundary of the UIelement 2130 is illustrated by the solid lines, the inset region 2110 isillustrated by the innermost dotted lines, the outset region 2120 isillustrated by the outermost dotted lines, and the radius of the digitindicators is shown by lines 2140.

The process identifies (at 2030) a hotspot for each digit indicator ofthe MT pointer. The hotspot for each digit indicator is the center pointwithin the digit indicator's hotzone. Each digit indicator hotspot isassociated with a single x-y coordinate. The position of each digitindicator hotspot is then checked (at 2040) to determine if any digitindicator is unambiguously positioned over a UI element. The followingpresents the various scenarios that may exist when performing UI elementdisambiguation and the result of the disambiguation process 2000. Asshown in FIG. 22, when the hotspot 2210 of a particular digit indicator2215 falls within an inset region of a UI element 2220, that particulardigit indicator 2215 is unambiguously positioned over that UI element2220 and therefore that particular digit indicator 2215 can be activatedto access that UI element 2220. As shown in FIG. 23, when the hotspot2310 of a particular digit indicator 2315 falls within an outset regionof only a single UI element 2320, that particular digit indicator 2315is unambiguously positioned over that UI element 2320 and therefore thatparticular digit indicator 2315 can be activated to access that UIelement 2320. As shown in FIG. 24, when the hotspot 2410 of a particulardigit indicator 2415 falls within the outset regions of at least two UIelements 2420 and 2430, then that particular digit indicator 2415 isambiguously positioned and therefore cannot be used to activate eitherof the UI elements 2420 and 2430. Lastly as shown in FIG. 25, when thehotspot 2510 of a particular digit indicator 2515 falls within theoutset region of a first UI element 2520 and within the inset region ofa second UI element 2530, then that particular digit indicator 2515 isunambiguously positioned over the second UI element 2530 and can be usedto activate the second UI element 2530 only.

The above described UI element disambiguation process can be used forenabling different MTI functionality in addition to or instead ofdetermining if a digit indicator activation should occur. One such usageof the disambiguation process is automated UI element identification.Automated UI element identification is a process that is performed inconjunction with MT pointer movements to provide visual identifiers thatidentify when a particular digit indicator of the MT pointer can be usedto access a UI element. In some embodiments, the visual identifiersinclude outlining or in some way identifying or highlighting the UIelement that becomes accessible. Other visual, audible, or tactileidentifiers may also be provided in addition to or instead of theoutlining and highlighting. For example, the coloring, transparency, orother visual aspect of the digit indicators may change.

FIG. 26 illustrates the visual identifiers in accordance with someembodiments. This figure illustrates two different MT pointer positions2610 and 2620. At the first MT pointer position 2610, UI element 2630 ishighlighted because the hotzone of the index finger digit indicator 2640is unambiguously positioned over the UI element 2630. At the first MTpointer position 2610, the other digit indicators cannot be used toaccess other UI elements, because their hotzone positioning is ambiguousrelative to two or more different UI elements.

At the second MT pointer position 2620, the index finger digit indicator2640 can still be used to access the UI element 2630, as indicated bythe highlighting of the UI element 2630. Additionally, the middle fingerdigit indicator 2650 can now be used to access the UI element 2670 andthe ring finger digit indicator 2660 can now be used to access the UIelement 2680 as identified by the highlighting of the UI elements 2670and 2680. At the second MT pointer position 2620, the ambiguity as tothe positioning of the middle finger digit indicator 2650 and the ringfinger digit indicator 2660 has been resolved. This is because thehotzone of each digit indicator 2650 and 2660 is now positioned withinan acceptable non-ambiguous range of a particular UI element.

In conjunction with the UI element disambiguation and highlighting, someembodiments display badges adjacent to one or more of the digitindicators, wherein the badges convey information such as whether the UIelement has functionality associated with a left-click, right-click, ordouble-click operation. Such badges may be activated when the MT pointeris proximate to the object, or the object is select or activated, or byother means such as a gesture or press of one or more keys. Someembodiments display groups of badges simultaneously, and provide furtherinformation if then a badge is subsequently clicked or otherwiseactivated, and fade out when other badges are activated or after someperiod of inactivity.

d. MTI Auto-Snap

To improve the ease with which a user accesses different UI elements,some embodiments of the MTI provide an auto-snap function thatautomatically places the digit indicators of the MT pointer horizontallyover UI elements along a single y-axis dimensional plane of the GUI. They-axis dimensional plane can be moved up and down by the user using amouse, trackpad, keyboard, or other input device. Similarly, the mouse,trackpad, keyboard, or other input device can be used to slide the digitindicators laterally across the set of UI elements on the particulary-axial dimensional plane. Since the digit indicators are automaticallysnapped to a set of four neighboring UI elements and since the digitindicators may be moved with simple up, down, left, and right discreteinputs or gestures, the user can more readily access UI elements withless GUI navigation and, more importantly, with less reliance onprecision movements which require a high degree of fine-motor skills onthe part of the user.

In some embodiments of the MTI auto-snap function, the MT pointer isrepresented as a set of four horizontally linear digit indicators thatare partially transparent and color coded, and are connected by ahorizontal line which extends across the window and/or display. Asbefore, each digit indicator is associated with one of the index,middle, ring, and little fingers such that touches detected on atrackpad or other touch surface that are assigned to the index, middle,ring, and little fingers can be used to activate the corresponding digitindicator. The home keyboard keys are also usable to activate thecurrent set of four linear digit indicators, and in some embodimentsthere is an on-screen keyboard viewer which shows the color-coded keysthat correspond to the activation of the digit indicators.

FIGS. 27, 28, 29, and 30 illustrate operation of the MTI auto-snapfunction in accordance with some embodiments. FIG. 27 illustrates theMTI auto-snap function at a first y-axial plane of the GUI, FIG. 28illustrates the MTI auto-snap function at a second y-axial plane of theGUI, and FIG. 29 illustrates the MTI auto-snap function at a thirdy-axial plane of the GUI.

In FIG. 27, the digit indicators are shown as translucent circles 2710,2720, 2730, and 2740. The circles 2710, 2720, 2730, and 2740 areautomatically positioned over the first four accessible UI elements atthe y-axial plane denoted by the horizontal line 2750 connecting thedigit indicators 2710, 2720, 2730, and 2740. The UI element over whichdigit indicator 2710 is positioned can be accessed with a tap of theindex finger on the trackpad or by pressing the “J” keyboard key, the UIelement over which digit indicator 2720 is positioned can be accessedwith a tap of the middle finger on the trackpad or by pressing the “K”keyboard key, the UI element over which digit indicator 2730 ispositioned can be accessed with a tap of the ring finger on the trackpador by pressing the “L” keyboard key, and the UI element over which digitindicator 2740 is positioned can be accessed with a tap of the littlefinger on the trackpad or by pressing the “Semicolon” keyboard key.

The user can move the digit indicators 2710-2740 to the next y-axialplane as depicted by line 2810 in FIG. 28 by moving a touch on atrackpad downward by moving the thumb downward, by moving a mousedownward, by pressing the down arrow key (or equivalent key or keys) onthe keyboard, or by some other gesture. When the y-axial plane changes,the digit indicators automatically snap to the closest four UI elementson the next y-axial plane. The user can shift the digit indicators tothe right or left along the current y-axial plane by moving a touch on atrackpad sideways, by moving a mouse sideways, or by pressing the leftor right arrow keys (or equivalent keys) on the keyboard. For example,FIG. 30 illustrates the position of the digit indicators 2710-2740 alongthe y-axial plane of FIG. 29 after the digit indicators 2710-2740 weremoved two incremental positions to the right.

In this manner, the user can quickly access a set of four UI elementsalong a particular y-axial plane. As such, the user no longer has tomanually reposition the digit indicators of the MT pointer over UIelements. Moreover, the set of four UI elements need not be directlyadjacent to one another in order to be accessed using the MTIfunctionality.

FIG. 31 presents a process 3100 for performing the auto-snap MTIfunctionality in accordance with some embodiments. The process 3100begins by identifying (at 3110) a current x-axis and y-axis coordinateposition of the GUI pointer even though the pointer will be hidden fromthe screen when the MT pointer appears. In some embodiments, theposition of the MT pointer and its constituent digit indicators isindependent of the GUI pointer, and is stored per context within eachapplication. In such embodiments, step 3110 would not use the GUIpointer as the starting location, but instead use the prior MT pointerlocation for the current application and/or document. Next, the processdetermines (at 3120) which UI elements are located along a thresholdzone of a particular y-axis plane that is determined from the y-axiscoordinate position of the MT pointer. In some embodiments, the UIelement identification is made by querying the system for a list of UIelements that are currently on-screen, and analyzing them based on theiry-axis positions relative to the on-screen coordinates. The processobtains (at 3130) the coordinates for each UI element along theidentified y-axis plane. The process snaps (at 3140) the digitindicators to the four UI elements that are closest to the x-axiscoordinate position of the MT pointer. Internally, the MTI maintains thecurrent position of the MT pointer as a center point and each digitindicator position is determined as an offset from the leftmost (indexfinger) digit indicator where the offset accounts for the distance fromthe leftmost digit indicator to the identified UI element. As oneskilled in the art would realize in light of the present disclosure,there are many techniques and technologies to identify and position theMTI digit indicators, as well as ways to optimize UI element datafetching, sorting, etc. Such anticipated techniques will vary fromsystem to system, and are envisioned by the present invention.

In some embodiments, more than four digit indicators are displayed onscreen, especially in cases where a keyboard key or keys are used toactivate the digit indicators. This is because keys next to or above thehome keys can also be utilized, unlike when a trackpad is used foractivation and there is a limitation as to the number of fingers on theuser's hand. As such, in some embodiments, two or more y-axial planesmay be displayed concurrently, and in a certain mode all UI elementcenters may be temporarily identified on screen to aid the user in theiridentification. In some embodiments, optional horizontal and/or verticalindicators also appear on-screen to aid the user in locating the UIelements, and animation is used to, among other things, smoothlytransition between movements of the MT pointer on the x-axis and/or they-axis. Such horizontal and vertical linear indicators may in someembodiments be extended to include the entire screen, or only thefrontmost window. In some embodiments, the user may jump to UI elementsthat are not included in the frontmost window or frontmost application.When the user taps or activates such a non-frontmost application UIelement, the MTI or system switches to the application that is the ownerof the activated UI element. Due to the UI element query and analysisprocess described above, other attributes of a set of UI elements areknown by the MTI, and may be presented to the user through variousvisual, audible or tactile means, such as visual badges as describedabove with reference to the core MTI functionality.

II. Alternative MTI Enhancements

A. Hold Functionality

In some embodiments, the core MTI functionality is enhanced to provide“hold action” functionality. In some such embodiments, hold actionfunctionality is enabled when a particular MTI digit indicator isactivated and held beyond a certain time threshold. So long as thatparticular MTI digit indicator remains active, the corresponding holdaction functionality is enabled. To keep an MTI digit indicator activeand thereby keep the hold action functionality enabled, the user retainsthe touch of the finger that activates that particular MTI digitindicator on the trackpad or other device, or the user continues to holddown the keyboard key that activates that particular MTI digitindicator. While the hold action functionality is enabled, touches ofother fingers or other keyboard keys not used in enabling the holdaction functionality can be used to provide functionality unrelated tothe activation of the digit indicators. In other words, the touches ofother fingers or other keyboard keys (not used to enable the holdaction) do not manipulate other UI elements, but instead add furtheroptions and capabilities to the UI element that is selected or isuniquely identified beneath the MTI digit indicator that is activatedand held beyond the certain time threshold. For example, pressing andholding the “J” keyboard key to activate the index finger digitindicator of the MT pointer beyond a time threshold enables a holdaction functionality whereby the “K”, “L”, and “Semicolon” keyboard keysmay be used to invoke different functions unrelated to activating themiddle, ring, or little finger digit indicators of the MT pointer. Whenthe hold action functionality is disabled, the touches of the otherfingers or other keyboard home keys are again usable to activate theother digit indicators. In some embodiments, different sets of functionsbecome enabled when a different MTI digit indicator is activated toenable hold action functionality. In some embodiments, the hold actionfunctionality is adapted for use in conjunction with a single pointersuch that the hold action functionality modifies whichever UI elementthe single pointer is positioned over.

FIG. 32A illustrates various hold action functionality in accordancewith some embodiments. This figure illustrates different sets offunctions 3210 a, 3220 a, 3230 a, and 3240 a that become enabled when adifferent MTI digit indicator is activated to enable hold actionfunctionality. As shown at 3210 a, when the index finger digit indicatorof the MT pointer is activated, a first set of functions associated witha first hold action become enabled such that the touch (i.e., middlefinger) or keyboard key(s) (i.e., “D” or “K”) that normally activatesthe middle finger digit indicator can now be used to perform a “copy”function, the touch (i.e., ring finger) or keyboard key(s) (i.e., “S” or“L”) that normally activates the ring finger digit indicator can now beused to perform a “paste” function, and the touch (i.e., little finger)or keyboard key(s) (i.e., “A” or “Semicolon”) that normally activatesthe little finger digit indicator can now be used to perform a “cut”function.

MTI digit indicator 3220 a of FIG. 32A illustrates a different set offunctions that become invocable when a second hold action is enabled byactivating the middle finger digit indicator of the MT pointer. As shownat 3220 a, when the middle finger digit indicator of the MT pointer isactivated, touch (i.e., index finger) or keyboard key(s) (i.e., “F” or“J”) that normally activates the index finger digit indicator can now beused to perform a word selection function whereby directional trackpadslides and/or keyboard keys select text on a word by word basis, thetouch (i.e., ring finger) or keyboard key(s) (i.e., “S” or “L”) thatnormally activates the ring finger digit indicator can now be used toperform a sentence selection function whereby directional trackpadslides and/or keyboard keys select text on a sentence by sentence basis,and the touch (i.e., little finger) or keyboard key(s) (i.e., “A” or“Semicolon”) that normally activates the little finger digit indicatorcan now be used to perform a character selection function wherebydirectional trackpad slides and/or keyboard keys select text on acharacter by character basis. Using the set of functions, selections canbe made in one of two directions. Such directional control is accessibleusing directional flicks or taps on a trackpad and specified keys on akeyboard. For example, holding down a touch of the middle finger on atrackpad activates the middle finger digit indicator and enables thehold action functionality of 3220 a such that upwards flicks of theindex finger on the trackpad can be used to select words in a firstdirection and downwards flicks of the index finger on the trackpad canbe used to select words in a second direction opposite to the firstdirection. Similarly, holding down the “K” keyboard key also activatesthe middle finger digit indicator and also enables the hold actionfunctionality of 3220 a such that taps of the “U” keyboard key (that isabove the “J” keyboard key) can be used to select words in a firstdirection and taps of the “M” keyboard key (that is below the “J”keyboard key) can be used to select words in a second direction oppositeto the first direction. In some embodiments, keyboards that havetouch-enabled surfaces are able to respond to directional slides andflicks to modify the selection size, without need to depress additionalkeys.

MT digit indicator 3230 a illustrates a third set of functions thatbecome invocable when a third hold action is enabled by activating thering finger digit indicator of the MT pointer. In this case, whenholding the ring finger on an object, a subsequent tap of the index ormiddle finger causes a menu UI element associated with that object toappear. In some embodiments, this is equivalent to a mouse right clickon an object. Further taps of the index and middle fingers (or taps oncorresponding keys) changes the selected menu item in a corresponding upand down direction. In alternate embodiments, slides or flicks of theindex finger result in a corresponding menu item selection, and includeinertial scrolling for menus with many items. In hierarchical menus,slides and flicks left and right also navigate into and out of nestedmenu items. Keyboards that have touch-enabled surfaces may also navigatemenu items by sliding or flicking motions on the surface of the key,without necessarily depressing the key. When the ring finger isreleased, the output action described by the currently selected menuitem is performed and the menu UI element is dismissed. In otherembodiments, a flick of the originally held touch is required to executethe action of the menu item, whereas releasing the originally held touchdismisses the menu without performing any action. Similarly, a tap ofthe little finger cancels menu operation by dismissing the on-screenmenu UI element without performing any output action, but it allows thestill touching ring finger to move to a different location to discloseanother menu UI element.

MTI digit indicator 3240 a illustrates a fourth set of functions thatbecome invocable when a fourth hold action is enabled by activating thelittle finger digit indicator of the MT pointer. Similar to the previousexample, the index and middle fingers may be tapped or slid or flickedto change the value of the active or selected UI element, which could bea slider, scroll bar, numeric value, etc. A tap of the ring fingerresets the value to a default value, and subsequent taps return thevalue to previously set values.

It should be noted that the four examples given of MTI hold actionsrepresent only a few examples of the functionality and output actionsthat are supported by the MTI system. In fact, some embodiments allowthe user to create custom MTI digit displays with programmable outputaction assignments for each gesture received after invocation. Moreover,as noted above, MTI functionality can be adapted to be operable withenhanced keyboards that have more than one touch sensitive key. Forinstance, when each keyboard key is touch sensitive, flicks and slidescan be performed directly over different keys to invoke the various MTIfunctionality. In some embodiments, slides or flicks issued on each ofthe home keys “A”, “S”, “D”, and “F” can be used to perform scrolling indifferent scroll areas. Specifically, flicking upwards on the “A” touchsensitive key performs an upward scroll in a first scroll area andflicking downwards on the “S” touch sensitive key performs a downwardscroll in a second scroll area. Scrolling ballistics and inertialscrolling algorithms are used to adjust the rate at which the scrollingoccurs based on the velocity with which the slide or flick is issued.Similarly, slides or flicks on other keys can be used to move orreposition the MT pointer. In some such embodiments, the entire keyboardsurface is transformed into a large continuous trackpad surface.

It should be noted that the hold actions described above do notnecessarily require positioning the MT pointer atop any particular UIelement. For example, when invoking the cut/copy/paste output actionsdescribed in 3210 a, the user may touch and hold their index fingeranywhere on the trackpad, touchscreen or other device, and if thatfinger is not moved beyond a distance and no other fingers begintouching within a particular time period, the MT digit display appears,which alerts the user to the new functionality available by manipulatingnew touches with the other fingers or thumb. In this case, tapping themiddle finger copies the currently selected text, regardless of thelocation of the fingers. In some embodiments such as MT digit display3230 a, the user may use the ring finger to select or hover over aparticular UI element, and then press and hold the ring finger to invokethe described functionality that is specific to the particular UIelement over which the pointer or ring finger is located. In otherwords, the functionality of some MTI hold actions utilize the positionof the initial holding touch (touchscreen) or the current pointerposition (trackpad), and other MTI hold actions are global in nature andare not location or pointer specific, and may be referred to asomni-touch in nature. Further detail about such location-independent UIis provided in the nonprovisional patent application Ser. No. 13/423,212with reference to omni-touch.

FIG. 32B provides an example of both location specific and omni-touchfunctionality while performing the common task of selecting some text,and then moving that text to another location within the paragraph. Theleft column 3200 b depicts a simplified view of the MTI digit display ofsome embodiments, and the right column 3210 b depicts a text field witha possible cursor position or selection within it. In step 1, the usertaps the index finger at the location of the word “sunlit” 3220 b, whichmoves the cursor to that location. On a touchscreen, the user wouldsimply tap at that location. Next, the user holds down the middle finger(at any location) to invoke the selection hold action 3220 a. Oneinvoked, the user taps the index finger to select the current word 3230b. The user then moves the index finger up to select the previous word3240 b as shown in step 3. Next, the user moves the middle finger in adesignated direction, to change the functionality of subsequent holdactions, and the new auxiliary output actions are interactivelypresented on the MTI digit display. In this case, a downward movementcauses subsequent words to be added to the selection rather thanselected individually. The user then slides the index finger down toselect multiple words 3250 b, and then flicks downward to further extendthe selection 3260 b. Because slides and flicks of the index finger aretied to inertial algorithms, the user is able to correlate the speed oftheir slides and flicks with the number of words selected. If too manywords are selected, a slide or flick in the opposite (up) directionselects fewer words. With the desired selection evident on screen, theuser releases the middle finger, thus exiting the select hold actions3220 a mode. Next, in step 5, the user presses and holds the indexfinger (at any location) to enable the copy/paste hold actions 3210 a,and taps the little finger, which causes the selected text 3260 b to becut, and removed from the screen. When the user lifts the index finger,the hold actions mode 3210 a is exited and the user may then use theindex finger to move the cursor normally (trackpad) or simply tap at thenext desired location (touchscreen). In this case, the user taps theindex finger just before the word “Through” as indicated by 2320 b toplace the cursor and paste insert location. Finally, the user once againenters the copy/paste hold actions 3210 a by holding the index fingerdown (at any location), and then tapping the ring finger to paste thepreviously cut text into the new location, as can be seen in theresulting text field 3280 b of step 7.

FIG. 32C details the process 3200 c used to determine various holdaction modes and operations. The process begins when a touch is detectedat 3210 c after a short period of time (e.g., 200 ms) during which notouches were down. In some embodiments a single touch is required tobegin the process, and in other embodiments multiple touches, when beganessentially simultaneously, are used as the basis of the hold action.This example refers to the single finger base hold action in thesingular for clarity, though multiple initial touches are equallyapplicable. If the original touch is moved beyond a short distance(e.g., 2 mm) within a time period (e.g., 1.5 s) 3220 c, the processexits 3225 c without activating any hold action mode. Otherwise, ifadditional touches are detected 3230 c, the process also exits 3235 cwithout activation. In some embodiments times and distances whichdetermine activation and other aspects of the MTI system are dynamicallyadjusted per user and context according to usage patterns and detectionof unintended actions. If the original touch is released 3240 c within atime period (e.g., 1.5 s) the process exits 3245 c without activation.If, however the touch is still being held, the heads-up display 3250 cof the MTI digit display appears on screen. Next, additional touches3260 c are used to invoke corresponding output actions 3265 c, bymonitoring and timing various manipulations of the additional touches todetermine taps, holds, slides, flicks, etc. and mapping a tokenizedrepresentation of that input to a database which stores the appropriateoutput action information, which may include system or API calls,scripts, or other means of effectuating some action or change on thedevice. Furthermore, if the original touch is moved 3270 c beyond athreshold distance (e.g., 10 mm) within a time period (e.g., 1 s), aflag is set which may modify further output actions 3275 c. In theexample referenced above in FIG. 32B, a downward slide changedindividual word selection to contiguous word selection. Subsequentslides of the original touch can revert the output actions to theoriginal form, or may introduce other auxiliary modes to subsequentoutput actions. In some embodiments, further output actions are invokeddirectly by manipulations of the original touch and/or modified by theaverage speed and direction of any original touch movement. When theoriginal touch is released 3280 c, the HUD is removed from the screen3285 c and the process ends 3290 c. Otherwise, the process continuallymonitors touch manipulations 3260 c and 3270 c until the original touchis finally released 3280 c.

B. Tap-Hold Functionality

In some embodiments, the assignment of touches to specific digits (thumband fingers) may not be reliably accurate due to hardware limitations orother issues. The MTI system accommodates such limitations inembodiments that differ from the above described hold action activationby requiring a four finger touch or tap followed by the holding of aspecific digit or digits. FIG. 32D provides further detail on how thesetap-hold actions are invoked and operated. The process 3200 d beginswhen one or more touches 3210 d are detected after a short period oftime (e.g., 200 ms) during which no touches were down. Additionaltouches are accepted 3220 d within an overall period of time (e.g., 0.5s) to essentially coalesce the touches. If any touches were moved orreleased within that time 3230 d, the process ends 3235 d withoutactivation. Otherwise, if less than four touches were detected 3240 d,the process ends 3245 d without activation. Essentially, the processrequires four essentially simultaneous non-moving touches in order tocontinue. In some embodiments where the thumb is detected and tracked,five touches are accepted, but may trigger alternate output actions.When the conditions are met, the heads-up display 3250 d of the MTIdigit display appears on the screen to show available options based onfinger combinations. Next, the user releases one, two or three of thetouches 3260 d essentially simultaneously, leaving at least one touchdown to determine the tap-hold action basis. The remaining touch ortouches are held down until the user chooses to exit the tap-hold actionmode. As described above in FIG. 32C, additional touch manipulations3270 d invoke corresponding output actions 3275 d, and manipulations ofthe held touch or touches 3280 d may trigger additional output actionsand/or set auxiliary modes 3285 d. When the remaining touches arereleased 2890 d, the HUD is removed from the screen 3295 d, and theprocess ends 3299 d.

FIG. 32E depicts a simplified HUD of the MTI digit display 3200 e. Inthis embodiment, ten icons (3211 e-3241 e) animate from the top of thescreen when the HUD first appears 3250 d. The top row (3211 e, 3212 e,3213 e, 3214 e) represents the base tap-hold mode that will result ifthat single finger is held (and the other touches released). HUD 3200 eis shown in right hand mode, so the top row icons represent (left toright) the index, middle, ring and little fingers. The remaining iconsrepresent combinations of multiple fingers, which will be furtherexplained later in this section. In some embodiments, HUD 3200 e appearswhen the user touches four fingers anywhere on the touch surface. Theuser can than identify (or remember) that the index finger representsthe desired “media player” application 3211 e, so the user releases allbut the index finger. At this point, the HUD changes 3260 d to the mediaplayer controls 3250 e, so that the user can identify (or remember) thevarious available options.

In practice, tap-hold invocations may be effectuated very quickly, sincethere are no perceptible minimum hold times required for activation andinvocation of the first output action. For example, 3250 e depicts atap-hold MTI digit display which is accessed when four fingers touch andthen all but the index finger are released. Essentially, all digits tapsimultaneously, except the index finger remains held down. These touchesmay occur anywhere on the sensing surface and are thus omni-touch innature, so the gesture can be performed very rapidly. As long as theinitial four touches are articulated in an essentially simultaneousmanner, an MTI-enabled OS will not accidentally trigger output actionsthat are directly above UI elements such as buttons. To further speedpractical operation, the user does not need to wait for the initial HUDdisplay 3200 e or the subsequent HUD display 3250 e to appear if theyhave already developed muscle memory for the desired functionality.

Continuing with the above example, the icon representing the indexfinger 3251 e is highlighted to show that this touch is held and is thebasis for subsequent output actions. For example, if the user next tapsthe ring finger 3253 e, an output action is generated which causes themedia player application to play the current track. To switch to thenext track the user taps the little finger 3254 e. The user may thenadjust the volume by sliding the middle finger 3252 e up or down. Theuser may then release the index finger 3251 e to leave the track playingand resume with other operations. When the user wants to again controlthe media player application, another tap-hold gesture leaving the indexfinger down re-invokes the HUD and allows the user to, for example, fastforward or rewind by sliding or flicking the ring finger 3253 e up ordown, switch to a different track by sliding or flicking the littlefinger 3254 e up or down, or stopping playback by tapping the middlefinger 3252 e. In some embodiments, combinations of fingers acting inconcert can trigger yet further output actions. The HUD 3250 e displayssuch combinations elsewhere on the HUD as a reminder to the user. Forexample, if the user taps the middle and ring fingers essentiallysimultaneously 3262 e, the media player announces the title of thecurrent track. If the user essentially simultaneously slides or flicksthe ring and little fingers 3263 e, the playback speed is altered. Ifthe user essentially slides or flicks the middle, ring and littlefingers together 3272 e, the media player selects a new playlist.Further details of such a triangle UI are provided in the nonprovisionalpatent application Ser. No. 13/423,212.

Further operational examples of the tap-hold functionality of the MTIsystem demonstrate the speed and ease of invoking commonly used outputactions. For example, a four finger tap ending with the middle and ringfingers held down controls the system volume 3222 e of some embodiments.After this invocation, the user simply slides the middle and ringfingers in essentially up and down directions to control the volume ofthe device, releasing the touches when the desired volume is attained.Furthermore, a tap of the index finger mutes the system volume, and atap of the little finger un-mutes. A new user will discover availablecommands via the HUDs, which can be resized by retaining the initialfour finger touch and sliding in essentially up down directions. As thedisplay is resized, more or less information such as icons,abbreviations, full descriptions and other information are presented inthe HUD as appropriate. In this manner, the user may discover newfunctionality easily and in-context, and frequently utilized controlsautomatically become part of muscle memory so that the HUD can besmaller in size, or simply ignored by the user.

In some embodiments, the MTI system determines the assigned digit basedon the locations of previously established four touch patterns, as wellas any drift that may have occurred by the remaining finger moving in aparticular direction. Details of the implementation of this system arecovered in the nonprovisional patent application Ser. No. 13/423,212.

C. Single Pointer Enhancements

In some MTI embodiments where multiple simultaneous pointers are notdesired or applicable, some MTI operations disclosed above may beutilized with a traditional single pointer or an enhanced version of thesingle pointer paradigm. This section highlights the single-pointer modeMTI functionality in which MTI functionality is adapted for usage with atraditional pointer or an enhanced version of the single pointer. Thissingle-pointer mode may be more appropriate for certain users or certainapplications or contexts which do not require or benefit from multiplesimultaneous pointers (i.e., digit indicators). Many of the abovedescribed embodiments, however, are applicable to this single-pointermode, which is activated by holding down a key other than the spacebarin some embodiments, and which may be represented on-screen as a singletranslucent circular icon which animates down from a larger size whenactivated. The animation, as shown in FIG. 33, serves to draw the user'seye to the location of the pointer, which is especially helpful onlarger screens, or if the pointer is over a text field and is a verythin “i-beam” cursor. Furthermore, unlike existing GUI pointer art whichutilizes a single x-y coordinate “hotspot” to distinguish UI elementsbelow, the MTI single-pointer mode employs the above described “hotzone”methodology and UI element disambiguation in some embodiments tofacilitate easier selection of UI elements. Movement of the MT pointerin single-pointer mode may also employ certain keys or combinations ofkeys to move the MT pointer, as described above.

In the MTI single-pointer mode of operation, holding down a specifiedkeyboard key beyond a specified threshold enables single-pointer modeMTI functionality and displays the single-pointer style MT pointer withthe beginning animation. When the pointing tool is placed over a UIelement and the single-pointer mode MTI functionality is enabled (e.g.,holding down the “V” key), pressing and holding different keys of thekeyboard invoke different functionality. For example, in someembodiments, the “F” keyboard key acts like the left mouse button inthat a tap can select the UI element at the location of thesingle-pointer MT pointer, and double-tapping the “F” key acts just as adouble-click. Similarly, when the “F” key is held down, a mouse-downevent is sent to the UI element and subsequent pointer movements causethe selected UI element to be dragged to a different on-screen positionwhile the “F” key remains held down. When the “F” key is released, amouse-up message is sent to the UI element and the dragging operationends. In a likewise manner, the “D” keyboard key may activateright-click functionality.

In some embodiments, various keyboard keys may be used in conjunctionwith the single-pointer mode of the MTI to facilitate scrolling withinone or more scroll areas of a UI. For example, pressing and holding the“S” keyboard key when the single-pointer mode is active enablesscrolling such that subsequent mouse or trackpad slides or flicksperform scrolling of a UI element at the location of the pointer whilethe “S” key remains held down. In some embodiments, different keyboardkeys may be mapped to activate scrolling within different scroll areasof a specific window, application, or UI. For example, enabling MTIfunctionality and then pressing and holding the “U” key activatesscrolling in a first scroll area, whereas enabling MTI functionality andthen pressing and holding the “I” key activates scrolling in a secondscroll area.

FIG. 33 illustrates single-pointer mode MTI functionality in accordancewith some embodiments. In this figure, the MT pointer is enabled insingle-pointer mode via spacebar down or other keyboard key down. The MTpointer is then positioned over one of the various UI elements 3310,3320, 3330, 3340, 3350, 3360, and 3370 using traditional means ofpointer movement (e.g., trackpad, mouse, arrow or directional keyboardkeys). The UI elements 3310-3370 each display and control a range ofspecific values. Once positioned over a particular UI element of the setof UI elements 3310-3370, the “A” key 3380 is pressed and held to enterthe value adjust mode. In this mode, the remaining home keys are used toquickly adjust the value of whichever UI element the MT pointer isunambiguously positioned over (i.e., highlighted). For example, the “F”keyboard key 3395 can be used to increment the value of the UI elementover which the MT pointer is positioned, the “D” keyboard key 3390 canbe used to decrement a value of the UI element over which the MT pointeris positioned, and the “S” keyboard key 3385 can be used to set adefault value for the UI element over which the pointing tool ispositioned. Also, by holding down the “F” or “D” keys in someembodiments, the value is continuously incremented or decremented.Furthermore, the rate of the increment or decrement may vary dependingon the type of controller or the range of values for the controller.Also, when the opposing increment key (e.g., “D” when “F” is down, or“F” when “D” is down) is pressed in some embodiments, the rate at whichthe values change is increased to allow the user to reach distant valuesmore quickly.

These single-pointer mode MTI functionalities are not intended to belimiting in scope. Accordingly, other alternative sets of MTIfunctionality may be provided in different embodiments. Moreover, thesealternative sets of MTI functionality may be adapted to work with thecore MTI functionality by mapping the alternative MTI functionality todifferent keyboard keys than the home keys of the keyboard and byfurther adapting the alternative MTI functionality to operate inconjunction with the MT pointer.

The MTI and its various permutations described in sections I and IIabove are applicable to any computing system or electronic device thatutilizes a pointing tool or GUI interface. As such, the MTI is operableon desktop computing systems, laptops, netbooks, and other portabledevices which contain a processor. Moreover, the MTI can be operatedusing any of a plurality of input devices separately or in combinationwith one another. As some examples, the MTI can be operated with just atraditional keyboard, with an enhanced keyboard with at least one touchsensitive key, a traditional keyboard and trackpad combination, atraditional keyboard and mouse combination, a traditional keyboard witha touchscreen, a trackpad, a touchscreen, or an input device associatedwith operation of a media center or “ten foot” interface. Furthermore, adevice such as a tablet, smart phone, media player, or other computingdevice may serve as an input device to a different device upon which theMTI is functioning.

III. Synchronized Multiple Device UIs

Human physiology inescapably maintains that the eyes are some distancefrom the hands, and therefore a single physical device that tries toaccommodate both hands and eyes will ultimately fail ergonomically. Theuser may accommodate the device for a short period of time by eitherbending the neck down to look toward the hands on a horizontal plane, orraising the hands up to meet the gaze of the eyes on a vertical plane.While this accommodation by the user may be convenient for short-termportable use, longer term use at some point will leave the user withchronic physical ailments.

It is therefore an objective of the present invention to provide theuser with visual aids and other tools which enable the brain to make astrong correspondence between hand/finger motions and resulting computeroperations, in order to minimize or eliminate the need for the user toconstantly shift visual and manual focus. It is further an objective ofthe present invention to define an apparatus, method, system, andcomputer software product that permits and encourages the device user toretain visual focus on the display device(s) on a vertical plane andmanual touch on the input device(s) on a horizontal plane. To achievethe objectives, some embodiments provide a multi-device UI thatcomprises at least two UIs, wherein a first horizontal UI is touchenabled and the first horizontal UI displays at least a subset of thesecond vertical UI. The horizontal UI may be presented using atouchscreen tablet, touchscreen smartphone, or other touch sensitivedisplay on an essentially horizontal plane that is synchronously coupledwith the vertical UI that may be presented using a monitor or displaydevice that may be placed farther from the user on an essentiallyvertical plane. In some embodiments, the one or more UIs aresynchronously coupled using a communication protocol such as UniversalSerial Bus (USB), WiFi, Bluetooth, or any other suitable protocol.Synchronously coupling the horizontal UI to the vertical UI allows touchinputs issued using the horizontal UI to control and produce change onthe vertical UI. Moreover, when the horizontal UI is embodied in atablet, smartphone, or other touch sensitive portable device, the datafrom both UIs can be stored to the memory of the tablet, smartphone, orother touch sensitive portable device. In this manner, the data can becarried with the user using a portable storage medium.

FIG. 34A presents a usage example of the synchronized multi-device UIsin accordance with some embodiments. The figure includes a first displaydevice 3410 a on an essentially horizontal plane displaying a first UIand a second display device 3420 a on an essentially vertical planedisplaying a second UI. The horizontal display device 3410 a is touchsensitive such that user touches can be used to provide inputs to thesecond UI 3420 a. As shown, the horizontal UI 3410 a displays a subsetof the vertical UI 3420 a. Specifically, the vertical UI 3420 a includesmultiple windows with a particular window that is frontmost, and thehorizontal UI 3410 a displays only the frontmost window of the verticalUI 3420 a. However, by synchronously coupling the UIs, touch inputsperformed on the horizontal UI 3410 a are usable to control operationson the vertical UI 3420 a. In this figure, the horizontal UI 3410 aprovides window switching functionality whereby a touch or flick in onedirection switches the frontmost window of both UIs in a first directionand a touch or flick in the opposite direction switches the frontmostwindow of both UIs in a second direction that is opposite to the firstdirection. Additionally, UI elements 3430 a appearing in top row of thehorizontal UI 3410 a represent each of the windows in the vertical UI3420 a, and touching any of the UI elements 3430 a causes thecorresponding window in the vertical UI 3420 a to become frontmost.Should a different window become frontmost, the display of thehorizontal UI 3410 a changes in accordance.

FIG. 34B presents an alternative synchronized multi-device embodimentwherein the horizontal device 3450 b is used to control variousoperations on the vertical device 3460 b. The horizontal device 3450 bhas a center area 3456 b which shows three miniature windows 3451 b,3452 b and 3453 b, that correspond with actual windows 3461 b, 3462 band 3463 b on the vertical device 3460 b. The miniature version of thefrontmost window 3453 b may be highlighted in a variety of ways, in thiscase by a dotted rectangle. The miniature windows 3456 b on thehorizontal device are smaller versions of the actual windows, and notmeant to be read by the user. Instead, the user may perform a variety ofuseful operations by manipulating the miniature windows on thehorizontal device 3450 b and immediately see the result reflected on thevertical device 3460 b. For example, the user may tap a window's icon(in the row above 3455 b) to bring the actual window frontmost, the usermay slide a miniature window's icon to move the position of the actualwindow, the user may use a pinch gesture in a miniature window to zoomthe contents of the actual window, the user may drag a border of aminiature window to resize the actual window, the user may slide orflick within a miniature window to scroll the contents of the actualwindow, etc. In the example illustrated in FIG. 34B, the user is workingwith two word processing documents 3461 b and 3462 b, and the user isalso referencing a web browser 3463 b. The user may bring the middlewindow frontmost by tapping on the corresponding middle applicationdocument icon displayed at the top row 3455 b of the horizontal device3450 b. This allows the user to view and compare the two documents sideby side. The user may then scroll the documents independently ortogether by sliding or flicking atop either or both of the miniaturewindows 3451 b and 3452 b. Other operations and contextual options areavailable in the lower row 3457 b of the horizontal device 3450 b.

FIG. 34C shows the process used by some embodiments to implement thesynchronized multi-device functionality of FIG. 34B. The OS of thevertical device is made aware of the presence of the horizontal devicewhen a communication protocol (e.g., USB, WiFi, Bluetooth, etc.)establishes (at 3410 c) the connection between devices. Variousattributes of the horizontal device are then communicated (at 3415 c) tothe vertical device, such as screen size and other capabilities. Systemstate information is sent (at 3420 c) to the horizontal device, thenreceived (at 3430 c) by the horizontal device. Such system stateinformation may include a list of windows and their positions, sizes andordering, a miniaturized graphic for each of the windows, frontmostapplication, focus of frontmost application, selection, etc. Someembodiments accommodate a range of protocols for packaging such systemdata. For instance, raw pixel data may be sent, but more efficientprotocols which have a shared understanding of the underlying objectprimitives or high level commands may be utilized for greaterefficiency, flexibility, and control possibilities. The horizontaldevice processes the received information from the vertical device andupdates the application (e.g., window contents, window positions, etc.)to reflect the new system state information. The vertical deviceconstantly monitors (at 3425 c) such system state information and sends(at 3420 c) it to the horizontal device. The horizontal device monitors(at 3440 c) user touch articulations to determine user intent. Thehorizontal device then generates (at 3445 c) corresponding commands in aprotocol which can be understood by the vertical device. The generatedcommands are sent (at 3450 c) to the vertical device via the appropriateprotocol, and received (at 3455 c) by the horizontal device. Thevertical device makes (at 3460 c) the corresponding changes to theappropriate objects. In some embodiments, the newly received commandscarry out the intended functionality, but do not send the correspondingsystem state information back to the horizontal device in order topreserve bandwidth. In other embodiments, the system state informationupdates are sent to the horizontal device to confirm that the commandswere properly interpreted.

FIG. 35A illustrates an alternate usage example of the synchronizedmulti-device UIs in accordance with some embodiments. In this figure,the horizontal UI 3510 a provides a first view of a frontmostapplication and the vertical UI 3520 a provides a second view of thefrontmost application where the first view includes a subset ofinformation from the horizontal view that is more focused or zoomed-in.The user may utilize the controls on the left side of the horizontaldevice 3510 a to switch to a different view (e.g., week view) in orderto zoom to a different view which utilizes slides to change the timingof upcoming events. The first view (e.g., month view) on the verticaldevice 3520 a does not necessarily change with the horizontal device3510 a, as the user may prefer to grasp the overall context on thevertical device 3520 a (which has a larger display), and edit thedetails (e.g., week or day view) on the horizontal device 3510 a. If thedata exists (in this case calendar data) on both devices, the data isupdated in both data stores as changes are made, along with anycloud-based storage update. If the horizontal device 3510 a isdisconnected from the vertical device 3520 a, upon re-connection anyedits are updated on the vertical device 3520 a.

FIG. 35B presents an alternative synchronized multi-device embodimentthat correlates with the embodiment presented in FIG. 34B above. In theprevious example, if the user double taps an application icon on thehorizontal device 3450 b, a detail view is presented on the horizontaldevice 3450 b. The user can easily switch back and forth between thewindow view as shown in FIG. 34B, and a focus view shown in FIG. 35B,which is the opposite in that it displays an easily editable view of thefocus of the frontmost application of the vertical device 3540 b. Thevertical device 3540 b displays the same three windows presented in 3460b, and also includes a view port rectangle shown as a dotted line in thefrontmost word processing document 3541 b. The text within the view portis reflected in the focus view area 3532 b of the horizontal device 3530b. When the user changes the text on the vertical device 3540 b, thetext is updated on the horizontal device 3530 b, and likewise when theuser edits the text on the horizontal device 3530 b it appears incontext on the vertical device 3540 b. This provides multiple views intothe same data (the word processing document), and allows the UI ofeither device to be utilized to its best advantage. For example, whenselecting and moving paragraphs within a document, the larger size andpoint and click interface of the vertical device 3540 b may be moreadvantageous, and when composing text, the small portable device 3530 bmay be preferable. When the connection between devices becomes available(such as WiFi proximity), the device with the latest version is used,and then edited synchronously until such time when the connection is notavailable. The focus mode of the multi-device synchronized UI isapplicable in many situations and workflows where there is a need towork on a detailed portion of a larger work, while simultaneouslyallowing a larger overall view of the work to be understood in context.For example, a music scoring application would display the score whichshows all the instruments in context on the vertical device, and thehorizontal device displays only the specific instrument that is beingedited in a large, easy to edit format. Furthermore, such edits can bemade using a more intuitive touchscreen UI where gestures may bepreferred over the single point and click UI of most traditionalvertical devices. When the vertical device is MTI-enabled, thehorizontal device can also be used to perform such gesture like commandson the vertical device as well. The area of focus displayed on thehorizontal device is a continuously scrolling view in some embodiments,which interactively updates the viewport and pagination of the viewpresented on the vertical device. The focus itself can be changed oneither the horizontal or the vertical device. For example, when the userclicks on the middle window 3542 b to bring that word processingdocument frontmost on the vertical device 3540 b, the focus view of thehorizontal device 3530 b is changed to reflect the location of thecursor or selection or previous viewport area of that document. Someembodiments of the synchronized multi-device UI support transfer ofother types of data to facilitate extended features such astext-to-speech. When the user performs a particular gesture on thehorizontal device 3530 b, the selected text is sent to the verticaldevice 3540 b along with a command to speak the selected text. Thisallows the vertical device 3540 b (which is usually connected to a powersource and has better audio capabilities) to render and playback theselected text, while preserving CPU cycles and thus battery life on thehorizontal device 3530 b. Similarly, if a video editing application isbeing used in a synchronized multi-device embodiment, the controls (suchas jog, shuttle, effect settings, etc.) may be invoked by the user withconvenient gestures, at hand level, on the horizontal device 3530 b, andthe CPU intensive rendering and display of the resulting operations ishandled by the vertical device 3540 b. In such a setting, the user maychoose to import a lower resolution or compressed version of a projectto the horizontal device 3530 b for viewing when the vertical device3540 b not connected.

FIG. 36A illustrates still another alternate usage example of asynchronized multi-device UIs in accordance with some embodiments. Inthis figure, the horizontal device 3610 a displays tools for interactingwith a workspace that is presented using the vertical device 3620 a. Byproviding the tools on the horizontal device 3610 a, the tools areaccessible by touch even though the user may retain most visual focus onthe vertical device 3620 a where the tools are used to interact with theworkspace.

In some embodiments, the communication protocol coupling the two devicesallows the UI of the touch enabled device to extend the UI of theprimary display. Accordingly, various UI elements or windows can bedragged from one UI to the other and the communication protocol keepsthe two UIs synchronized.

FIG. 36B presents an alternative synchronized multi-device embodimentwherein the screen graphics of the horizontal device 3630 b are sent tothe vertical device 3640 b so that they can be used for reference.Because many touchscreen devices have small screens, one of thelimitations of their use is the difficulty of referring to work orinformation contained within another application. To resolve this issue,some embodiments send a representation of the bit map image of thehorizontal device 3630 b to the vertical device 3640 b, where previousbit map representations can also be seen for reference. As illustratedin 3641 b, the frontmost application appears in the upper left corner,the previous application 3643 b appears in the upper right corner, theapplication previous to that 3642 b appears in the lower left corner,and the application previous to that 3644 b is represented in the lowerright corner. As the user changes applications on the horizontal device3630 b, the representations on the vertical device 3640 b shift to showthe current application as well as the previous three applications. Insome embodiments, the current application on the horizontal device 3630b is not shown on the vertical device 3640 b so that live updates of thecurrently running application do not need to occur on the verticaldevice 3640 b, and instead the previous four applications arerepresented. Some embodiments support a varying number of screenrepresentations, as well as zoom functionality. Other embodiments enablethe interactions on the vertical device 3640 b to send messages to thehorizontal device 3630 b to, for example, switch to the selectedapplication. It should be noted that in some embodiments the horizontaldevice 3630 b drives the video to a vertical device, so that anon-computer device such as a TV monitor can be used as the verticaldevice via protocols such as WiFi or even a hard-wired video connection.

IV. Precision Pointer Interface

A common issue confronting users of trackpad, touchscreen, or othertouch sensing input devices is the inability to perform preciseselections and movements, because the human finger is a fairly thickpointing device that does not come to a single point. Touchscreensfurther exacerbate the issue because the touch that the user utilizes tomake the selection or movement obscures the on-screen area at which aprecise selection or movement is needed. Accordingly, it is an objectiveof the present invention to define an apparatus, method, system, andcomputer software product that overcomes limitations and enhances theuse of the human finger as a UI pointing device on touchscreen andtrackpad devices. Specifically, some embodiments provide a precisionpointing function that seamlessly operates with traditional trackpadand/or touchscreen functionality.

In some embodiments, the precision pointer includes a configurablemagnified region to better represent underlying on-screen content. Thelevel of magnification adjusts the speed of movement over the underlyingcontent and with the magnified view of the underlying content, the useris able to more precisely position the pointer. For a touchscreen, themagnified region is presented at the edge of the user's touch such thatthe user's touch does not obscure the underlying content.

In some embodiments, the user performs a two touch gesture to enable theprecision pointer functionality. One touch of the two touch gesture mayinclude a touch of the thumb. The inclusion of the thumb allows foreasier detection of the gesture, because the thumb produces arecognizable touch pattern that is distinguishable from the touches ofthe index, middle, ring, and little fingers. Moreover, the inclusion ofthe thumb allows for greater dexterity when performing contraction andexpansion gestures to adjust the amount of magnification. The figuresbelow illustrate operation of the precision pointer based on touchesthat are applied to a trackpad and not a touchscreen so as to clearlydifferentiate the touch actions from the results they produce. However,the precision pointer is implemented such that the same touch actionscan also be applied to a touchscreen device in order to produce the sameresults and the same resulting functionality.

FIG. 37 illustrates the precision pointer in accordance with someembodiments. As shown, the precision pointer 3710 is enabled with a twotouch gesture on the trackpad where the gesture includes a touch of thethumb 3720 and one other finger 3730, usually the index finger, in anessentially vertical orientation. In some touchscreen embodiments, whenan above-described two touch gesture is located near the top or bottomof the touchscreen, the precision pointer is presented to the side ofthe touches, rather than above where there may not be sufficient screenspace to display the precision pointer. The left/right designation maychange depending on whether the left or right thumb was used, and/or ifthe touches are close to the left or right side of the touchscreen.

Once enabled, the precision pointer 3710 is displayed on-screen in placeof a traditional GUI pointer. The user moves the precision pointer 3710on-screen by moving the two touches 3720 and 3730 in an essentiallyparallel manner, or by removing one of the two touches and then movingthe remaining touch. When both fingers are lifted, the precision pointerremains on-screen for some time period to allow for a tap gesture by afinger. If no tap is registered within a specified time duration, theprecision pointer fades from the screen, without any functionalityinvoked, and normal operation resumes. In some embodiments, a tap of thethumb after all touches are released causes another action, orimmediately dismisses the precision pointer. Likewise if a finger tapoccurs in a substantially new x-y coordinate, then no action is takenand the precision pointer is dismissed immediately, with an animation orother feedback in some embodiments. If a slide touch by either theoriginating thumb or finger occurs after all touches are released in asubstantially new x-y location, the precision pointer jumps to thatlocation and tracks the slide movements in some embodiments.

The precision pointer 3710 is represented by at least one circularregion through which the underlying on-screen content is visible. Therepresentation of the precision pointer 3710 as a circular region is notintended to be limiting and can therefore be substituted with othershapes or representations whereby similar functionality may beaccessible. In some embodiments, a digit indicator 3740 is also shown inaddition to the precision pointer, and in some embodiments, animation isemployed on invocation to draw the eye to the screen area of theprecision pointer. In some embodiments, other feedback mechanisms suchas sound and tactile response are employed to confirm to the user theinvocation and other modes of the precision pointer interface.

The level of magnification for the content visible through the circularregion of the precision pointer 3710 is user specifiable by expansionand contraction gestures of the two touches 3720 and 3730. When onetouch is used to move the precision pointer 3720, a second touch can bereintroduced at any time to perform the magnification adjustment wherethe two touches used to perform the magnification need not include atouch of the thumb. To increase the level of magnification, the userperforms an expansion gesture to increase the distance between the twotouches. To decrease the level of magnification, the user performs acontraction gestures to decrease the distance between the two touches.The amount of magnification is determined based on the distance withwhich the two touches are expanded or contracted.

FIG. 38 illustrates an increasing of the magnification level of theprecision pointer 3710 in response to an expansion gesture in accordancewith some embodiments. As shown on the trackpad, the two touches 3720and 3730 are moved away from one another so as to increase the distancebetween the two touches 3720 and 3730. In so doing, the contentdisplayed through the precision pointer 3710 is magnified to provide alarger view of the underlying content. The zoomed in view allows a userto more precisely target underlying content with the precision pointer3710 than with traditional pointing tools. In some embodiments, whenunderlying content becomes accessible (e.g., selectable or invocable), across-hair or other visual identifier is presented with the precisionpointer 3710 to notify the user. Moreover, the snapping functionality ofthe MTI can be used in conjunction with the precision pointer 3710 suchthat a UI element is automatically centered within the precision pointer3710 when that UI element partially intersects with a boundary of theprecision pointer 3710. With use of the precision pointer, trackpads andtouchscreens can be adapted for use with software applications thatrequire precise operations, such as graphical editing and media editingsoftware as some examples. When a precision pointer embodiment isintegrated into the OS of the device, it is operable across allapplications, without the need for individual software applicationmodifications.

FIG. 39 presents a process 3900 for implementing the precision pointerin accordance with some embodiments. The process 3900 is performed whenthe precision pointer is enabled. As noted above, the precision pointeris enabled using a two touch gesture or other touch gesture.

Once the precision pointer is enabled, the process 3900 begins byidentifying (at 3910) the current position of the on-screen pointer.Often, the on-screen pointer is represented as an arrow or I-beam withthe current position of the arrow pointing to one pixel or a set ofpixels at the tip of the arrow and with the current position of theI-beam identifying one pixel or a set of pixels at the vertical edge ofthe I-beam. In place of the on-screen pointer, the process displays (at3920) the precision pointer. As shown in FIGS. 37 and 38, the precisionpointer is displayed with an adjustable magnified region and mayoptionally display a digit indicator adjacent to the adjustablemagnified region. When displaying the precision pointer, the processtemporarily alters the representation of the on-screen pointer from theGUI/OS default representation (e.g., pointer or I-beam) to that of theprecision pointer. In some embodiments, the process issues one or moreGUI/OS system calls or API calls to change the representation of theon-screen pointer to that of the precision pointer.

Next, the process identifies (at 3930) the pixels that fall within theadjustable magnified region of the precision pointer. In someembodiments, the pixels include those that are some radius away from theposition of the on-screen pointer or are some other distance from theposition of the on-screen pointer. The process identifies (at 3940) amagnification level for the precision pointer. The magnification levelis a setting that determines how much magnification to apply to theunderlying region of the GUI that falls within the adjustable magnifiedregion of the precision pointer. In some embodiments, the precisionpointer is always enabled with a default magnification level. In someother embodiments, the precision pointer is enabled with themagnification level that was used when the precision pointer was lastenabled.

To actually magnify the underlying region of the GUI, the processdetermines (at 3950) an inset region of pixels within the identifiedpixels falling within the adjustable magnified region of the precisionpointer based on the identified magnification level. The inset region isidentified because some pixels falling along the outer region of theadjustable magnified region of the precision pointer will becomeobscured when magnifying those pixels that are in the inset region andthat are closer to the center of the precision pointer. For example,when there are 200 pixels falling within the diameter of the adjustablemagnified region of the precision pointer and the magnification level is2× (i.e., double), the inset region is determined to include theinnermost 100 pixel diameter of the 200 pixel diameter image. Theprocess then uses well-established graphical resizing algorithms toenlarge the innermost 100 pixel diameter image to produce a 200 pixeldiameter image that consumes the entire adjustable magnified region ofthe precision pointer thereby providing a two fold increase in size forGUI content represented by the innermost 100 pixel diameter image whileobscuring the outermost 100 pixel diameter image. Accordingly, theprocess magnifies (at 3960) the inset region of pixels to fill theadjustable magnified region of the precision pointer.

The process 3900 may continue as long as the precision pointerfunctionality is enabled. The process 3900 may continue to allow foradjustments to the magnification level for the GUI content displayed inthe adjustable magnified region of the precision based on expansion andcontraction gestures or other user input. For instance, when themagnification level is increased with an expansion gesture, the insetregion of pixels is redefined to include a smaller set of innermostpixels within the adjustable magnified region of the precision pointer.Those innermost pixels are then scaled up to produce an even greatermagnified representation for a more constrained region of the underlyingGUI content that otherwise falls within the adjustable magnified regionof the precision pointer. Similarly, when the magnification level isdecreased with a contraction gesture, the inset region of pixels isredefined to include a larger set of innermost pixels that are thenscaled to display a greater amount of the underlying GUI content withinthe adjustable magnified region of the precision pointer.

It should be apparent to one of ordinary skill in the art that at anymagnification level of the precision pointer, the underlying GUI contentthat is accessible or that can be invoked by the precision pointer doesnot change. This is because the precision pointer points to the same UIelement or underlying GUI content irrespective of the magnificationlevel, the magnification level merely assists the user in more preciselyvisualizing and isolating a desired UI element or underlying GUIcontent. In some embodiments, other digits are used after enabling theprecision pointer to access further functionality. In accordance withthe above usage of the MT pointer, different digits and touch gesturescan be used to perform hold, copy, and paste functionality as someexamples and which are detailed above.

V. Computer System

Each of the MTI, multi-UI, and precision pointer are implemented as oneor more software applications or processes that execute on one or morephysical computing devices. These applications may be specified as a setof instructions recorded to a non-transitory computer readable storagemedium (also referred to as computer readable medium) thereby providingan article of manufacture. When these instructions are executed by oneor more computational element(s) (such as processors or othercomputational elements like ASICs and FPGAs), they cause thecomputational element(s) to perform the actions indicated in theinstructions. Computer and computer system are meant in their broadestsense, and may include any electronic device with a processor includingcellular telephones, smartphones, portable digital assistants, tabletdevices, laptops, and desktops. Examples of computer readable mediainclude, but are not limited to, CD-ROMs, flash drives, RAM chips, harddrives, EPROMs, etc.

The MTI transforms general purpose computing resources of the computingdevices on which it executes to provide a particular machine with whichUI elements can be interacted with using the MT pointer and associatedfunctionality. The multi-UI transforms general purpose computingresources of the computing devices on which it executes to provide aparticular machine whereby the user's focus and operational focus neednot be on the same UI. The precision pointer transforms general purposecomputing resources of the computing devices on which it executes toprovide a particular machine with which UI elements can be preciselyinteracted with using an imprecise touch. Stated differently, thecomputing devices on which any of the MTI, multi-UI, and precisionpointer executes comprise general purpose processors, random accessmemory, non-volatile storage, and network resources that are transformedby any of the MTI, multi-UI, and precision pointer into one or morespecific purpose machines that perform associated function.

FIG. 40 illustrates a computer system with which some embodiments areimplemented. Such a computer system includes various types of computerreadable mediums and interfaces for various other types of computerreadable mediums that implement the various processes, modules, andengines described above for each of the MTI, multi-UI, and precisionpointer. Computer system 4000 includes a bus 4005, a processor 4010, asystem memory 4015, a read-only memory 4020, a permanent storage device4025, input devices 4030, and output devices 4035.

The bus 4005 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of thecomputer system 4000. For instance, the bus 4005 communicativelyconnects the processor 4010 with the read-only memory 4020, the systemmemory 4015, and the permanent storage device 4025. From these variousmemory units, the processor 4010 retrieves instructions to execute anddata to process in order to execute the processes of the invention. Theprocessor 4010 is a processing device such as a central processing unit,integrated circuit, graphical processing unit, etc.

The read-only-memory (ROM) 4020 stores static data and instructions thatare needed by the processor 4010 and other modules of the computersystem. The permanent storage device 4025, on the other hand, is aread-and-write memory device. This device is a non-volatile memory unitthat stores instructions and data even when the computer system 4000 isoff. Some embodiments of the invention use a mass-storage device (suchas a magnetic or optical disk and its corresponding disk drive) as thepermanent storage device 4025.

Other embodiments use a removable storage device (such as a flash drive)as the permanent storage device Like the permanent storage device 4025,the system memory 4015 is a read-and-write memory device. However,unlike the storage device 4025, the system memory is a volatileread-and-write memory, such as random access memory (RAM). The systemmemory stores some of the instructions and data that the processor needsat runtime. In some embodiments, the processes are stored in the systemmemory 4015, the permanent storage device 4025, and/or the read-onlymemory 4020.

The bus 4005 also connects to the input and output devices 4030 and4035. The input devices enable the user to communicate information andselect commands to the computer system. The input devices 4030 includeany of a capacitive touchscreen, resistive touchscreen, any othertouchscreen technology, a trackpad that is part of the computing system4000 or attached as a peripheral, a set of touch sensitive buttons ortouch sensitive keys that are used to provide inputs to the computingsystem 4000, or any other touch sensing hardware that detects multipletouches and that is coupled to the computing system 4000 or is attachedas a peripheral. The input device 4030 also include alphanumeric keypads(including physical keyboards and touchscreen keyboards), pointingdevices (also called “cursor control devices”). The input devices 4030also include audio input devices (e.g., microphones, MIDI musicalinstruments, etc.). The output devices 4035 display images generated bythe computer system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD).

Finally, as shown in FIG. 40, bus 4005 also couples computer 4000 to anetwork 4065 through a network adapter (not shown). In this manner, thecomputer can be a part of a network of computers (such as a local areanetwork (“LAN”), a wide area network (“WAN”), or an Intranet, or anetwork of networks, such as the internet. For example, the computer4000 may be coupled to a web server (network 4065) so that a web browserexecuting on the computer 4000 can interact with the web server as auser interacts with a GUI that operates in the web browser.

As mentioned above, the computer system 4000 may include one or more ofa variety of different computer-readable media. Some examples of suchcomputer-readable media include RAM, ROM, read-only compact discs(CD-ROM), recordable compact discs (CD-R), rewritable compact discs(CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layerDVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM,DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards,micro-SD cards, etc.), magnetic and/or solid state hard drives, ZIP®disks, read-only and recordable blu-ray discs, any other optical ormagnetic media, and floppy disks.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention.

I claim:
 1. A method of interfacing with a computer using multipledigits, the method comprising: detecting a touch from a first digit thatdoes not move past a threshold distance during a specific hold duration;suppressing a first set of output actions based on said detecting,wherein the first set of output actions comprises at least one outputaction invoking in response to a touch or tap issued independent ofcontinued retention of said first digit touch; changing from the firstset of output actions to a different second set of output actions basedon said detecting and continued retention of said first digit touch,wherein the second set of output actions comprises at least one outputaction invoking in response to a tap or touch from a second digit issuedin conjunction with continued retention of said first digit touch; andexecuting a first output action from the second set of output actionsbased on retention of the first digit touch and a tap or touch of thesecond digit.
 2. The method of claim 1 further comprising executing asecond output action from the second set of output actions based on saidretention of the first digit touch and a tap or touch of a third digitdetected some threshold distance from the location of the second digittap or touch.
 3. The method of claim 1 further comprising executing asecond output action from the second set of output actions based on saidretention of the first digit touch and a tap or touch of a third digitdetected some threshold distance from the location of the retained firstdigit touch.
 4. The method of claim 1 further comprising restoringaccess to said first set of output actions in response to a release ofall touches, and wherein said executing of the first output action isnot required to restore said access.
 5. The method of claim 1 furthercomprising changing from the second set of output actions to a differentthird set of output actions in response to said retention of the firstdigit touch and sliding the first digit touch in a particular directionby at least a specific distance.
 6. The method of claim 1 furthercomprising retaining the first set of output actions in response to saidretention of the first digit touch and sliding the first digit touchpast the threshold distance during the specific hold duration.
 7. Themethod of claim 1 further comprising selecting the second set of outputactions from different sets of output actions based on the first digittouch being performed by a first detected digit of the different digits.8. The method of claim 7 further comprising selecting a different thirdset of output actions from the different sets of output actions inresponse to the first digit touch being performed by a different seconddigit of the different digits.
 9. The method of claim 1 furthercomprising executing a second output action from the second set ofoutput actions based on retention of the first digit touch and adirectional slide or flick of the second digit.
 10. The method of claim1, wherein said detecting the first digit touch comprises receiving afirst input of a press and hold of a first keyboard key, and whereinsaid executing the first output action comprises receiving a secondinput of a tap of a second keyboard key while continuing to receive saidfirst input.
 11. The method of claim 1, wherein said detecting the firstdigit touch comprises detecting the first digit touch at a locationproximate to a first location, and wherein said executing the firstoutput action comprises detecting the second digit tap or touch at alocation proximate to a second location which is different from thefirst location.
 12. A method of interfacing with a computer using tapand hold touches, the method comprising: enabling access to differentsets of output actions in response to first input comprising a pluralityof touches issued essentially simultaneously and held on a touch inputdevice for at least a specific duration; changing the first input tosecond input in response to removal of a first set of the plurality oftouches and retention of a different second set of the plurality oftouches, wherein the first set of touches comprises at least a firsttouch of the plurality of touches and the second set of touchescomprises at least a different second touch of the plurality of touches;selecting from the different sets of output actions based on whichtouches of the plurality of touches are removed as part of said removalof the first set of touches and which touches of the plurality oftouches are retained as part of said retention of the different secondset of touches.
 13. The method of claim 12, wherein said selectingcomprises enabling a first set of output actions from the different setsof output actions in response to the first set of touches removing aleftmost touch of the plurality of touches and the second set of touchesretaining a rightmost touch of the plurality of touches, and furtherenabling a different second set of output actions from the differentsets of output actions in response to the first set of touches removingthe rightmost touch of the plurality of touches and the second set oftouches retaining the leftmost touch of the plurality of touches. 14.The method of claim 12, wherein said selecting further comprisesenabling a first set of output actions from the different sets of outputactions in response to the first set of touches removing a first touchand a second touch of the plurality of touches and the second set oftouches retaining a third touch of the plurality of touches, and furtherenabling a different second set of output actions from the differentsets of output actions in response to the first set of touches removingthe first touch and the second set of touches retaining the second touchand the third touch.
 15. The method of claim 12, wherein said selectingcomprises enabling a particular set of output actions in response to thefirst set of touches removing at least a first touch of the plurality oftouches, the method further comprising invoking an output action of theparticular set of output actions in response to detecting the firsttouch being reintroduced on the touch input device with a tap while thesecond set of touches remain held down.
 16. The method of claim 15,wherein invoking the output action comprises invoking a first outputaction of the particular set of output actions in response to detectingthe first touch being reintroduced above a location of the first touchissued as part of said enabling access with the plurality of touches,and invoking a different second output action of the particular set ofoutput actions in response to detecting the first touch beingreintroduced below the location of the first touch issued as part ofsaid enabling access with the plurality of touches.
 17. The method ofclaim 12 further comprising presenting on a display, a user interfacewith a particular set of output actions from the different sets ofoutput actions based on said selecting enabling the particular set ofoutput actions.
 18. A method comprising: adjusting a position of apointer in a graphical user interface (GUI) based on user input to afirst input device; invoking a first pointer operation from the positionof the pointer in the GUI in response to a manipulation of a first keyon a second input device; and invoking a different second pointeroperation from the position of the pointer in the GUI in response to amanipulation of a different second key on the second input device. 19.The method of claim 18, wherein said invoking the first pointeroperation comprises performing a mouse left-click operation in responseto a tap of said manipulated first key, and wherein said invoking thesecond pointer operation comprises performing a mouse right-clickoperation in response to a tap of said manipulated the second key. 20.The method of claim 18, wherein said invoking the first pointeroperation comprises performing a mouse left-button down operation inresponse to the first key being pressed down, performing a mouseleft-button up operation in response to a release of the first key afterthe first key is pressed down, and performing a mouse left-button dragin response to pointer movements with the first input device that occurbetween said first key being pressed down and said release of the firstkey.